Interactive editing of electronic text

Abstract

The new segment of electronic handwriting is provided to a handwriting recognition module to obtain a plurality of textual interpretations of the new segment. The textual interpretations obtained from the handwriting recognition module are scored based on how each electronic handwriting representation would change the display of the existing electronic content if the electronic handwriting representation were substantially displayed at a user-specified location within or near the existing electronic content. Based on the score, an electronic handwriting representation corresponding to a corresponding textual interpretation in the plurality of textual interpretations is selected, and the existing electronic content is modified to include the selected electronic handwriting representation substantially located at the user-specified location.

Description

Background Art

[0001] Editing of existing electronic documents is traditionally done through traditional keyboard and mouse input. For example, a user can select a portion of a document to be edited and provide input via a keyboard to erase, add or modify the content in the document. Some software applications allow the use of a stylus to add handwritten annotations to a document. However, in these applications, the electronic handwriting applied to the document is usually static. Either the input is permanently displayed as electronic handwriting, or the electronic handwriting is converted as a whole into character-based content. If a mistake is made, the user must undo the previous action and redraw the annotations in the document. In other words, the editing of the document is completed after the entire document is converted into a computer-based text form, and all user-generated content will eventually be displayed in text form. Summary of the invention

[0002] Aspects of the subject technology provide systems and methods for providing interactive editing of electronic text by using electronic handwriting. The purpose of electronic handwriting is to encourage creative and free-form hybrid layouts, in which graphics, drawings, and text can be placed anywhere in the document. To this end, the subject technology provides a computer-implemented method, the method comprising: receiving a new segment of electronic handwriting for display at a user-specified location within or near existing electronic content displayed on a display device; providing the received new segment to a handwriting recognition module to obtain a plurality of text interpretations of the new segment; generating corresponding electronic handwriting representations for the plurality of text interpretations obtained from the handwriting recognition module; scoring the plurality of text interpretations obtained from the handwriting recognition module based on how each corresponding electronic handwriting representation will change the display of the existing electronic content when the corresponding electronic handwriting representation is substantially displayed at the user-specified location within or near the existing electronic content; selecting an electronic handwriting representation corresponding to a corresponding text interpretation in the plurality of text interpretations based on the scoring; and modifying the existing electronic content to include the selected electronic handwriting representation substantially located at the user-specified location for display on the display device. Other aspects include corresponding systems, apparatus, and computer program products for implementing the computer-implemented method.

[0003] Further aspects of the subject technology, features and advantages, as well as the structure and operation of various aspects of the subject technology are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various aspects of the subject technology are described with reference to the accompanying drawings. In the drawings, like reference numbers may indicate identical or functionally similar elements.

[0005] Figure 1An exemplary system for editing electronic text in accordance with aspects of the subject technology is shown.

[0006] Figure 2A and Figure 2B Depicted is an example handwriting recognition interface displayed on a touch display in accordance with aspects of the subject technology.

[0007] Figure 3 is an example rearrangement of characters input into a display in accordance with aspects of the subject technology.

[0008] Figure 4 Depicting an example tree of content actions for recognizing electronic handwriting in accordance with aspects of the subject technology.

[0009] Figure 5 An example process for editing electronic handwriting in accordance with aspects of the subject technology is depicted.

[0010] Figure 6 is a conceptual diagram illustrating an example electronic system 600 for recognizing and editing electronic handwriting in accordance with aspects of the subject technology. DETAILED DESCRIPTION

[0011] Although various aspects of the subject technology are described herein with reference to illustrative examples for specific applications, it should be understood that the subject technology is not limited to those specific applications. Those skilled in the art will recognize additional modifications, applications and aspects within its scope, and additional fields in which the subject technology will have important uses through access to the teachings provided herein.

[0012] There is a need to preserve user-generated content in handwriting and to allow interactive editing of electronic handwriting content. There is also a need to have a user interface that allows interaction between computer-based text and handwriting while maintaining the natural layout of written documents.

[0013] The subject technology provides a software application that a user can interact with a structured document, which includes existing electronic content consisting of a string of electronic characters and / or electronic handwriting generated by the user (e.g., one or more lines of text or paragraphs, graphics or symbols). The electronic content can be decomposed into multiple segments or blocks for dynamically editing the content. Each segment or block can be the smallest definable unit of text (such as a character or word), or can contain a series of handwriting strokes or one or more continuous strokes. A segment can include characters, partial characters, strikethroughs, lines, scribbles, underlines, insertion symbols, boundary shapes, etc. Therefore, when a user writes, all strokes in a document are automatically clustered into blocks. The grouping may take into account the physical proximity of the strokes, the time when the strokes were created, etc.

[0014] By using a pointing device such as a stylus or other touch device, a user can add a series of strokes to a document at a user-specified location to represent a word or some other electronic handwriting annotation. The word or stroke can modify or be added to existing electronic content. The system responds to a new handwriting interaction by generating all possible document interpretations of the interaction, and ranks the document interpretations. In this regard, when new segments of electronic handwriting are received, they are provided to the handwriting recognition module 141. The handwriting recognition module 141 analyzes each segment and generates a textual interpretation of the segment. In this regard, the textual interpretation can include or take into account segments including characters, partial characters, strikethroughs, lines, scribbles, underlines, insertion symbols, boundary shapes, etc. The handwriting recognition module 141 typically generates multiple textual interpretations.

[0015] To generate the interpretation, the parser dynamically generates a content action tree for the newly received electronic handwriting segment. Each lower node branching from a higher level node in the tree represents one of a plurality of content actions that can be performed on the existing electronic content to identify a portion of the existing electronic content for associating with at least a portion of the new segment. For example, a content action can include identifying an existing paragraph or a line of existing text to which the new segment can be assigned.

[0016] The identified portion of the existing electronic content may be depicted according to a plurality of nodes along a given branch. For example, a first node in a branch may identify a paragraph to which a newly received segment may be assigned, and then a second node in the branch may identify a line within the paragraph to which the received segment may be assigned. Thus, the algorithm automatically traverses the tree along corresponding paths from the root node of the tree to the various leaf nodes of the tree to identify candidate portions of the existing electronic content to be provided to the handwriting recognition module 141 along with the new segment. Each determined candidate portion is then provided to the recognition along with the newly received electronic handwriting segment to obtain a plurality of textual interpretations of the new segment.

[0017] The text interpretation generated by the handwriting recognition module 141 can then be utilized by a handwriting synthesizer that generates a corresponding handwriting representation of each of the plurality of text interpretations obtained from the handwriting recognition module 141 based on an analysis of the existing electronic handwriting in the document. Each of these electronic handwriting representations can include one or more computer-copied glyphs substantially in the handwriting style of the existing electronic content (e.g., replicating the text interpretation). These handwriting representations are then applied to the content of the existing document in a hidden background process and scored based on how each corresponding electronic handwriting representation changes the display of the existing electronic content when substantially displayed at a user-specified location within or near the existing electronic content.

[0018] The score of the handwriting interpretation may be based on, for example, a comparison of a visualization of the document including the original electronic handwriting as annotated by the user and a visualization of the document after the interpreted handwriting representation has been applied to the document (e.g., inserted into the electronic content). For example, if adding the interpreted handwriting representation results in an additional line in a paragraph or causes an additional paragraph to be added, the score may be reduced. If the boundaries of the paragraph or line in which the interpreted handwriting representation was added remain substantially the same, the score may remain the same or increase.

[0019] The system then selects the best interpretation based on the ranking and updates the document accordingly. The existing electronic content is then modified to include the selected electronic handwriting representation substantially at the user-specified location and displayed on a display device. As the user writes, the document is reformatted and the spacing is updated in real time. Additionally or alternatively, the user can delay updating the document until the edit is complete and then manually update the document by selecting an update control.

[0020] In some implementations, electronic handwriting representations may not be generated (or desired), and scoring may be performed based on multiple text interpretations obtained from the handwriting recognition module 141. In this regard, the system may convert electronic handwriting received from a user, convert the handwriting into electronic text, and dynamically insert the newly converted electronic text into existing electronic content after selecting a scored text interpretation.

[0021] Feedback loop

[0022] As described above, after receiving a section of electronic handwriting and updating the document, the system begins to monitor additional feedback from the user to determine whether the insertion of the selected electronic handwriting representation can be considered to be accepted or rejected by the user. Acceptance is indicated by the user taking no further action on the newly inserted handwriting representation. However, rejection can be indicated by the user's modification or revocation of the representation. If rejection is determined, the system can select a second electronic handwriting representation corresponding to one of the text interpretations previously provided by the handwriting recognition module 141. For example, the system can select the interpretation corresponding to the next highest score. In some embodiments, the text interpretation obtained from the handwriting recognition module 141 can be re-scored based on user input associated with the indication that the selected electronic handwriting representation is rejected. For example, the user can only cross out a portion of the handwriting segment of the new annotation, or add to it. Then, the updated handwriting segment (together with the selected portion of the existing electronic content, as described above) can be sent back by the handwriting recognition module 141, and a new score is generated based on the updated newly generated handwriting representation inserted into the document.

[0023] If the system cannot obtain a certain confidence level for the new handwriting input, the system can visually solicit feedback from the user. After scoring the text interpretation, the system can determine that the score does not meet the predetermined confidence level, and then visually identify a portion of the modified electronic content corresponding to the text interpretation as not meeting the predetermined confidence level. For example, a portion of the electronic handwriting representation applied to the document can be highlighted. Additionally or alternatively, before modifying the existing electronic content, the system can visually identify the location within or near the existing electronic content where the handwriting representation will be placed, and then prompt the user to accept or reject the specified location.

[0024] Additional Features

[0025] The system maintains the electronic content as both the electronic handwriting and the character-based text representation of the existing electronic content. When the electronic handwriting representation is added to the document, the character-based text representation is also updated to include the corresponding text interpretation. The system may also include an option to display the electronic handwriting simultaneously with the character-based text representation, so that each word of the character-based text representation is provided for display adjacent to or directly below a corresponding displayed word segment of the electronic handwriting.

[0026] The foregoing features, which will be described further below, provide a technical effect of enabling interaction between computer-based text and handwriting while maintaining the natural layout of a written document. In this regard, the subject technology enables a user to update a document simultaneously in both computer-generated text and electronic handwriting, without having to consider the particular mode in which the document should be edited for optimal results.

[0027] Figure 1 An exemplary system 100 for editing electronic text according to aspects of the subject technology is shown. System 100 includes computing device 101, touch device 178, network 170, and application server 180. Computing device 101 and / or touch device 178 may be coupled to application server 180 via network 170.

[0028] Computing device 101 includes processor 121, one or more input / output components (eg, display device 122), communication module 123, and memory 130. Memory 130 includes touch-enabled editor application 140, which may include handwriting recognition module 141 and scoring module 142.

[0029] In aspects of the subject technology, computing device 101 is a touch-enabled device that can receive touch input from a finger or other instrument via display device 122. For example, computing device 101 can be a desktop all-in-one computer, a tablet or convertible computer, a laptop computer, a smart phone, a portable game console, or other device with a display device 122 that supports electronic pen input.

[0030] In one example, touch input is generally used to refer to input from a touch device 178, which can be implemented as a finger, electronic pen, or stylus that can interface with a display device 122 of a computing device 101. For example, the touch device 178 can be an active device including processing circuitry, memory, and / or one or more applications stored in the pen device memory, or can be a passive device having a material composition that is configured to be detectable by circuitry within the display device 122. For example, a passive pen device can interface with a capacitive touch screen or a resistive touch screen, a display digitizer, a peripheral tablet device, or other input device coupled to or integrated with the display device 122. In certain aspects of the subject technology, the display device 122 can correspond to a display projector or augmented reality glasses, in which case the touch device 178 can be implemented as an air pen that can provide pen input without contacting a physical surface.

[0031] A user who wishes to handwrite text with a touch device 178 and computing device 101 may, for example, access an application store supported by application server 180 to download and install touch-enabled editor application 140 onto computing device 101. Thus, computing device 101 may communicate with application server 180 over network 170 using communication module 123 (e.g., an Ethernet adapter, a wireless card, or a cellular modem). In various aspects of the subject technology, touch-enabled editor application 140 may be provided on an installable medium (e.g., on an optical disk, flash memory card, or other device). In other aspects of the subject technology, touch-enabled editor application 140 may be pre-installed onto computing device 101, or touch-enabled editor application 140 may be integrated as part of an operating system of computing device 101.

[0032] The touch device 178 may interact with the touch-enabled editor application 140 via the display device 122 (e.g., by actively transmitting communications to electrodes or other receiving components in the display device, or by passively causing detectable changes (such as changes in capacitance or resistance in electrodes or other receiving components in the display device), and / or may interact with the touch-enabled editor application 140 by receiving and transmitting signals via the communications module 123.

[0033] Scoring module 142 can be included with touch-enabled editor application 140, or can be operably connected to application 140, such as as a local plug-in, or a web application located at a server and accessible via a web-based API. Touch-enabled editor application 140 can provide users with the ability to enter handwritten text and / or graphics and options for digitizing and / or clearing handwritten input after entry. Touch-enabled editor application 140 can also provide users with other types of input options, such as options for entering content from a keyboard, and / or options for inserting objects such as images, videos, or other drawings via a file browser or drag-and-drop operations.

[0034] One example function that may be performed by computing device 101 is recognizing handwritten text input to computing device 101. A user may input handwriting to computing device 101 for recognition by using a touch device 178 including a finger or stylus, mouse, trackball, pen, etc.

[0035] Handwriting input generally includes one or more strokes, such as a single point or a continuous line, which represent one or more characters in a writing system individually or collectively. A "stroke" can form a part of a character, one or more complete characters, a word or a part of a word, a multi-word phrase or even a complete sentence. A handwriting recognition module 141 such as a handwriting recognition module 141 parses the handwriting input into individual handwriting characters. The handwriting recognition module 141 determines text characters based on the handwriting characters respectively. The determined text characters are displayed to the user. The recognition can be a single character at a time, multiple characters at a time, one or more words at a time, or the recognition is less than a single character (a part of a single character). Therefore, when the description of this article discusses the recognition of handwriting input to determine a "character", it should be understood that the term "character" is intended to include a single character, multiple characters (including a word, multiple words, phrases, sentences, etc.), and a part of a single character.

[0036] Handwriting input recognition can be performed locally (at computing device 101 via display device 122), in the cloud (at a server connected to computing device 101 via a network), or a combination thereof. It should be understood that the techniques described herein as being performed at a "server" can be performed at more than one server in a distributed or parallel server arrangement. Therefore, the term "server" as used herein is intended to include a single server and multiple servers working in coordination.

[0037] In one or more embodiments, while recognizing handwriting input, possible stems of the handwritten words represented by the handwriting input, and possible whole words for one or more possible stems can be identified. The computing device 101 can display the possible stems to the user of the device. As will be further described, the recognition module 141 can automatically select one or more stems based on a scoring algorithm, and wait for further input from the user to confirm or reject the selection. In some embodiments, when one or more possible whole words have been identified for possible stems, the computing device 101 can display possible word indicators with possible stems. Then, the user can select the possible stems displayed together with the possible word indicators. When selecting a possible stem, the computing device 101 displays the possible words identified for the possible stems. The user can select one of the possible words or one of the possible stems before completing the recognition of the handwriting input.

[0038] If the user begins to enter another handwriting input while handwriting input is being recognized, computing device 101 can change the size of the first handwriting input passed to handwriting recognition module 141 and / or currently being recognized, for example, when the handwriting input has not yet been recognized. In some embodiments, computing device 101 can also display a recognition pending indicator to the user while handwriting input is currently being recognized and / or there are one or more additional handwriting inputs to be recognized.

[0039] Figure 2A and Figure 2B Depicts an example handwriting recognition interface displayed on a touch display 122 according to various embodiments. The editor application 140 displays a handwriting interface 202 on the touch display 122 for a user to input handwriting into the computing device 101 for recognition. The handwriting interface 202 includes a handwriting display portion 204, and the handwriting interface 200 may also include a whole or part of a keyboard, such as a QWERTY-based keyboard, a T9-based keyboard, or another suitable type of interface from which a user can enter spaces, carriage returns, punctuation marks, symbols, etc. In one example, the handwriting recognition interface is displayed on the touch display without an interface from which a user 108 can enter spaces, carriage returns, punctuation marks, symbols, etc.

[0040] like Figure 2AAs shown, in some embodiments, computing device 101 can generate two display parts on touch display 122: first display part 204 and optionally second display part 206. In the first display part, computing device 101 displays the handwriting input currently recognized and any additional handwriting input that has not yet been recognized. In the second display part, computing device 101 can display the text corresponding to the handwritten words and / or handwritten characters that have been recognized previously. In this regard, the text representation based on characters of the existing electronic content displayed on the display device is maintained in memory 130 and updated with the text interpretation of the handwriting representation selected by editor application 140. The text representation based on characters can be displayed in the second display part 206 or hidden from view.

[0041] The user can enter handwriting for recognition anywhere within the handwriting display portion 204, or in some embodiments anywhere on the touch display 122. The editor application 140 can display the handwriting input as the user enters the handwriting input in real time, during the handwriting recognition, or both. It should be understood that real time can include an acceptable amount of delay between input and display. As the handwriting input is recognized, the display of the text characters can provide the user with a visual indication of the progress of the recognition.

[0042] Once the user provides handwriting input, the user handwriting interface 200 can display the handwriting input in the handwriting display portion 204. The user can provide handwriting input by using cursive or printing (also known as print). Although handwriting in English is discussed and illustrated, the recognition and display techniques disclosed herein are also applicable to handwriting input in other languages.

[0043] In some implementations, an updated character-based text representation may be provided for display on a display device in a space adjacent to and / or below the input handwriting along with a modified version of the existing electronic content. Figure 2B As shown, an updated character-based text representation including a plurality of words, each word can be provided to be displayed adjacent to or directly below a corresponding displayed word segment of a modified version of existing electronic content. When a new stroke is recognized, the corresponding character-based text representation is displayed below the handwritten character as the handwritten character is input and is updated in real time.

[0044] Further references Figure 1 The handwriting recognition module 141 may parse the handwriting input into handwritten characters by using the data stored in the recognition data storage device 150. The editor application 140 may update the displayed content by using one or more handwritten characters to display a representation of the handwriting input.

[0045] Based on the data stored in the recognition data storage device 150, the handwriting recognition module 141 can recognize the handwritten characters of the handwriting input one by one. In other words, the handwriting recognition module 141 can determine the text character for the handwritten character one handwritten character at a time. In some embodiments, the handwriting recognition module 141 can recognize multiple handwritten characters of the handwriting input at one time. The order in which the handwriting recognition module 141 recognizes the handwritten characters can be a logical order of reading, such as from left to right for recognizing English.

[0046] The handwriting recognition module 141 may parse the handwriting input into various parts, such as one or more handwritten characters. For example, the handwriting recognition module 141 may parse the handwriting input "hello" into four handwritten characters that form the handwriting input. In various languages, characters may correspond to words.

[0047] Figure 3 is an example rearrangement of characters input into the display 122 according to one or more embodiments of the subject technology. The editor application 140 displays the text characters within the text display portion 204 as the text characters are determined to rearrange the existing content to incorporate the new strokes and the newly generated handwritten content. In the depicted example, the user enters the characters "butnot" via the touch display 122. The characters are passed to the handwriting recognition module 141, which parses the handwriting input into its components. Here, the handwriting recognition module 141 determines that the input characters include the two words "but" and "not", and returns two different versions of the phrase with different spaces inserted between the words. Each version is analyzed and scored along with the existing electronic content displayed in the handwriting interface 202.

[0048] refer to Figure 1 , the recognition of handwriting input can be performed in whole or in part by a remote server 180. The editor application 140 can capture a digital representation of the handwriting input input to the touch display 122. The communication module 123 can transmit the digital representation to the server 180 via the network 170. The network 170 can be, for example, a cellular network, a satellite network, or another suitable type of network. The server 180 may include a communication module, a handwriting recognition module 141, and a recognition data storage device 150. It should be understood that the server 500 may include additional computing components, such as one or more processors, memory, power supplies, and other suitable computing components.

[0049] Server 180 may receive the digital representation via communication module 123. Recognition module 141 may parse the digital representation into handwritten characters. Recognition module 141 may parse the handwritten input into handwritten characters, for example, by using data stored in recognition data storage 150. Server 180 may transmit one or more representations for the handwritten characters back to computing device 101 for computing device 101 to display the handwritten characters to a user (e.g., on display device 122).

[0050] The recognition module 141 can determine the text characters for the handwritten characters one at a time or multiple at a time. For example, the recognition module 141 can determine the text characters of the handwritten input by using the data stored in the recognition data storage device 512. The server 180 can transmit an indicator of the text characters for the handwritten characters to the computing device 101 when the text characters are determined. The computing device 101 determines the text characters based on the indicator and displays the text characters to the user via the touch display 122.

[0051] The digital representation captured by the editor application 140 is included in the request context to be sent to the recognition engine 142. In this regard, the request context may include the electronic handwriting stroke currently being input into the editor application 140 via the handwriting interface 202, as well as a set of already recognized characters preceding the stroke. The request context may also include a set of previously recognized characters following the stroke. For example, the editor application may detect that the user is attempting to insert a character between words or other characters.

[0052] Figure 4 Depicts an example tree of content actions for identifying electronic handwriting according to one or more embodiments of the subject technology. Recognition engine 142 includes a parser that generates a content action tree 400 for the current request context. The tree can be implemented as a data structure residing in memory 130. A parent node 402 is generated for one or more strokes. As previously described, each lower node 404 branching from a higher level node in the tree represents a content action in a plurality of content actions that can be performed with respect to existing electronic content to identify a portion of the existing electronic content for associating with at least a portion of a new segment. For example, a content action can include identifying an existing paragraph or a line of existing text to which the new segment can be assigned. Each line, paragraph, word, or character can be selected using a nearest neighbor algorithm and / or searched based on the strokes entered.

[0053] The identified portion of the existing electronic content can be depicted based on multiple nodes along a given branch. In the depicted example, the first node in the branch identifies an existing paragraph to which the newly received segment can be assigned, and the second node along the branch (e.g., a child of the first node) identifies the line in the paragraph to which the newly received segment can be assigned. The third node (a child of the parent node) identifies that the new stroke may belong to a new paragraph and assigns a new line to the stroke. Therefore, the algorithm automatically traverses the tree along the corresponding path from the root node of the tree to the corresponding leaf node of the tree to identify candidate portions of the existing electronic content to provide them to the handwriting recognition module 141 together with the new segment. Then, each determined candidate portion is provided to the recognition module with the newly received electronic handwriting segment to obtain multiple textual interpretations of the new segment. In one or more embodiments, the handwriting recognition module 141 can generate the tree and perform this task.

[0054] Figure 5 Describes an example process for editing electronic handwriting according to various aspects of the subject technology. For illustrative purposes, reference is made to Figures 1 to 4 Describe the various blocks of example process 500, and the components and / or processes described herein. One or more blocks in process 500 can be implemented, for example, by computing device 101, which includes a processor and other components utilized by device 101. In some embodiments, one or more blocks can be implemented separately from other blocks and implemented by one or more different processors or devices. Further for illustrative purposes, the blocks of example process 500 are described as occurring serially or linearly. However, multiple blocks of example process 500 can occur in parallel. In addition, the blocks of example process 500 do not need to be performed in the order shown, and / or one or more blocks of example process 500 do not need to be performed.

[0055] In the depicted example flow chart, the editor application 140 receives a new segment of electronic handwriting for display at a user-specified location within or adjacent to existing electronic content displayed on the display device 122 (502). For example, a user may electronically draw one or more strokes within the handwriting display portion 204 of the handwriting interface 200.

[0056] In response, the editor application 140 provides the received new segment to the handwriting recognition module 141 to obtain a plurality of text interpretations of the new segment (504). The handwriting recognition module 141 generates the text interpretations and passes them back to the editor application 140.

[0057] Before passing the received new segment to the handwriting recognition module 141, the editor application 140 can generate a content action tree for the new segment. As previously described, a root node can be first created for the new segment, and each lower node branching from a higher-level node (e.g., starting from the root node) can represent one of a plurality of content actions that can be performed on existing electronic content to identify a portion of the existing electronic content for associating with at least a portion of the new segment. By using this mechanism, the editor application 140 can automatically determine candidate portions of the existing electronic content to provide to the handwriting recognition module 141 along with the new segment without user intervention. The candidate portions are generated based on automatically traversing the tree along a corresponding path from the root node of the tree to a corresponding leaf node of the tree, and identifying portions of the existing electronic content associated with the content actions represented by the nodes along the path. Then, the new segment is provided to the handwriting recognition module together with each determined candidate portion of the existing electronic content to obtain a plurality of textual interpretations of the new segment.

[0058] It is worth noting that in some examples, the new segment of electronic handwriting received at the display 122 can modify the existing segment of electronic handwriting of the existing electronic content. In this case, the existing segment of electronic handwriting can be provided to the handwriting recognition module together with the new segment to obtain multiple text interpretations.

[0059] The editor application 140 then generates corresponding electronic handwriting representations (506) for the multiple text interpretations obtained from the handwriting recognition module 141. The electronic handwriting representation can be generated by using handwriting synthesis so that the electronic handwriting representation includes one or more computer-copied glyphs that are substantially in the handwriting style of the existing electronic content. The artificial handwriting generated by the editor application 140 can be based on the user's own handwriting samples that have been input into the editor application 140. The editor application 140 can match the user's style, font size, and stroke characteristics. In some embodiments, generating the electronic handwriting representation by the editor application 140 includes the editor application 140 forwarding the text interpretations to a handwriting synthesis engine (not shown).

[0060] The generated handwriting representation may not be (e.g., immediately) displayed to the user. In one or more embodiments, the editor application 140 passes the handwriting representation along with at least a portion of the existing electronic content to the scoring module 142. The editor application 140, in conjunction with the scoring module 142, assigns scores to the text interpretations obtained from the handwriting recognition module 141. Each text interpretation obtained from the handwriting recognition module 141 is scored (508) based on how the corresponding electronic handwriting representation of the text interpretation will change the display of the existing electronic content when the electronic handwriting representation is substantially displayed at a user-specified position within or near the existing electronic content. In this regard, the scoring module 142 can analyze the existing content with the inserted selected handwriting representation without display to determine how the existing content changes with the insertion. The score is calculated based on predetermined factors, such as changes to the following: word spacing, baseline alignment, character spacing, character deformation, number of lines on the current page, word or line or paragraph reflow, etc. For example, an increase or decrease in word spacing or word spacing may negatively affect the score proportional to the amount of increase or decrease. The score may be expressed as a value having a range (eg, from 0 to 1 or from -1 to 1), where 0 is a neutral or status quo value.

[0061] The editor application 140 then selects an electronic handwriting representation corresponding to a corresponding text interpretation from among the plurality of text interpretations based on the score (510). The editor application 140 modifies the existing electronic content to include the selected electronic handwriting representation substantially located at the user-specified location for display on the display device 122 (512). In this regard, the existing electronic content is modified to accommodate the selected electronic handwriting representation by dynamically re-formatting at least a portion of the existing electronic content.

[0062] In some embodiments, the editor application 140 displays one or more visual indicators of the progress of recognizing the handwriting input. For example, the computing device 101 may receive a handwriting (or character) representation corresponding to a first handwriting input and display the handwriting (or character) representation in addition to updating one or more visual indicators of the progress. The editor application 140 may receive a handwriting (or character) representation corresponding to a second handwriting input. The editor application 140 displays the handwriting (or character) representation corresponding to the second handwriting input and updates the visual indicator of the progress of the recognition. The editor application 140 may continue to receive handwriting (or character) representations (if any) (e.g., from the handwriting recognition module 141 or the server 180), display the determined text characters, and update the displayed visual indicators as the handwriting (or character) representations are received. Once all handwritten characters have been recognized, the editor application 140 may receive an indicator from the recognition module 141 (or the server 180) that the recognition of the handwriting input is complete.

[0063] In some embodiments, the visual indicator is enhanced to a different color or otherwise highlighted to signal that a predetermined confidence level (e.g., a threshold score) is not met. It can be determined that the score assigned to the selected handwriting representation does not meet the predetermined confidence level (e.g., if only one representation is returned). The editor application 140 can then visually identify the portion of the modified electronic content corresponding to the selected electronic handwriting representation as not meeting the predetermined confidence level.

[0064] In one or more embodiments, the editor application 140 includes a feedback loop that adjusts the scoring and selection of handwriting representations based on input (or lack of input) from the user. Before modifying the existing electronic content, the editor application 140 can visually specify a computer-specified location within or near the existing electronic content corresponding to the insertion point of the selected electronic handwriting. The editor application 140 monitors the user's indication of accepting or rejecting the selected electronic handwriting representation. Upon receiving the user's indication of rejecting the selected electronic handwriting representation, the editor application 140 selects the second electronic handwriting representation corresponding to the second text interpretation of the plurality of text interpretations as the selected electronic handwriting representation. If the rejected representation is the highest-scoring representation among the representations that are configured with scores based on data from the scoring module 142, the editor application 140 can select the representation with the next highest score. In some embodiments, the editor application 140 can re-score the representations as described above based on a new set of factors and select the highest representation. At the user's indication of rejection, the current representation is replaced with the new representation. The editor application 140 can consider, for example, the absence of a rejection indication after a predetermined period of time (e.g., 1 or 2 seconds) as an acceptance of the representation.

[0065] For example, an indication that a user rejected a selected electronic handwriting representation may be received. In response, the editor application 140 may select a second electronic handwriting representation corresponding to a second text interpretation in the plurality of text interpretations, and modify the existing electronic content to include the second electronic handwriting representation substantially located at a user-specified location for display on a display device. Prior to selecting the second representation, the editor application 140 may re-score the text interpretations obtained using the handwriting recognition module 141 based on user input associated with the indication that the selected electronic handwriting representation was rejected, and based on how each respective electronic handwriting representation would change the display of the existing electronic content if the respective electronic handwriting representation was substantially displayed at a user-specified location within or near the existing electronic content. The second electronic handwriting representation may then be selected based on the re-score.

[0066] Many of the above examples 500 and related features and applications may also be implemented as a software process, which is specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium) and can be automatically executed (e.g., without user intervention). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of processors, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0067] The term "software" is intended to include, where appropriate, firmware residing in a read-only memory or an application stored in a magnetic storage device that can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software aspects disclosed in the subject matter may be implemented as sub-parts of a larger program while retaining the different software aspects disclosed in the subject matter. In some embodiments, multiple software aspects may also be implemented as separate programs. Finally, any combination of separate programs that implement the software aspects described herein together is within the scope of the present disclosure. In some embodiments, a software program, when installed to operate on one or more electronic systems, defines one or more specific machine implementations for executing and performing the operations of the software program.

[0068] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file used to store other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files for storing one or more modules, subroutines, or partial codes). A computer program may be deployed to execute on one or more computers, the multiple computers being located on one site, or distributed across multiple sites and interconnected by a communication network.

[0069] Figure 6 5 is a conceptual diagram showing an example electronic system 600 for recognizing and editing electronic handwriting according to aspects of the subject technology. The electronic system 600 may be a system for performing one or more parts or steps of the process 500, or by Figures 1 to 4The electronic system 600 may be associated with a computing device that provides software associated with the components and processes. Figures 1 to 5 101. In this regard, the electronic system 600 or computing device 100 may be a personal computer or a wearable mobile device such as a smart phone, tablet computer, laptop computer, PDA, augmented reality device, such as a watch or band or glasses or a combination thereof, or other touch screen or television having one or more processors embedded therein or coupled thereto, or any other type of computer-related electronic device with network connectivity.

[0070] The electronic system 600 may include various types of computer-readable media and interfaces for various other types of computer-readable media. In the depicted example, the electronic system 600 includes a bus 608, a processing unit 612, a system memory 604, a read-only memory (ROM) 610, a permanent storage device 602, an input device interface 614, an output device interface 606, and one or more network interfaces 616. In some implementations, the electronic system 600 may include or be integrated with other computing devices or circuits for operation of the various components and processes previously described.

[0071] Bus 608 collectively represents all system buses, peripheral buses, and chipset buses used to communicatively connect the numerous internal devices of electronic system 600. For example, bus 608 communicatively connects processing unit 612 to ROM 610, system memory 604, and permanent storage device 602.

[0072] From these various memory units, processing unit 612 retrieves instructions to execute and data to process in order to perform the processes of the present disclosure. In various implementations, the processing unit may be a single processor or a multi-core processor.

[0073] ROM 610 stores static data and instructions required by processing unit 612 and other modules of the electronic system. On the other hand, permanent storage device 602 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 600 is turned off. Some embodiments of the present disclosure use a mass storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as permanent storage device 602.

[0074] Other embodiments use removable storage devices (e.g., floppy disks, flash drives, and their corresponding disk drives) as permanent storage devices 602. Like permanent storage devices 602, system memory 604 is a read-write memory device. However, unlike storage devices 602, system memory 604 is a volatile read-write memory, such as random access memory. System memory 604 stores some instructions and data required by the processor at runtime. In some embodiments, the processes disclosed in the subject matter are stored in system memory 604, permanent storage devices 602, and / or ROM 610. From these various memory units, processing unit 612 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

[0075] The bus 608 is also connected to input and output device interfaces 614 and 606. The input device interface 614 enables a user to convey information to the electronic system and select commands. Input devices used with the input device interface 614 include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 606 can, for example, display images generated by the electronic system 600. Output devices used with the output device interface 606 include, for example, a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices such as a touch screen that is used as both an input and output device.

[0076] In addition, if Figure 6 As shown, bus 608 also couples electronic system 600 to a network (not shown) via network interface 616. Network interface 616 may include, for example, a wireless access point (e.g., Bluetooth or WiFi) or a radio circuit for connecting to a wireless access point. Network interface 616 may also include hardware (e.g., Ethernet hardware) for connecting a computer to a portion of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), a wireless LAN, or an intranet, or a network such as the Internet. Any or all components of electronic system 600 may be used in conjunction with the present disclosure.

[0077] These functions can be implemented in computer software, firmware or hardware. The technology can be implemented by using one or more computer program products. Programmable processors and computers can be included in mobile devices or packaged as mobile devices. Processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through communication networks.

[0078] Some embodiments include electronic components, such as microprocessors, storage devices, and memories, which store computer program instructions in machine-readable media or computer-readable media (or referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), recordable compact disk (CD-R), rewritable compact disk (CD-RW), read-only digital versatile disk (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program executable by at least one processing unit and include sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing higher-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0079] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits (e.g., application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves.

[0080] As used in this specification and any claims of this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms "display" or "displaying" mean displaying on an electronic device. As used in this specification and any claims of this application, the term "computer-readable medium" is entirely limited to tangible physical objects used to store information in a computer-readable form. These terms exclude any wireless signals, wired download signals, and any other transient signals.

[0081] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0082] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser with which a user can interact with an implementation of the invention), or includes any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), inter-networks (e.g., the Internet), and point-to-point networks (e.g., ad hoc point-to-point networks).

[0083] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is due to the computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0084] Those skilled in the art will appreciate that the various illustrative frames, modules, elements, components, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, components, methods and algorithms have been generally described in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application. Without departing from the scope of the disclosed technology, various components and frames can be arranged differently (e.g., arranged in different orders or separated in different ways).

[0085] It should be understood that the specific order or hierarchy of steps in the disclosed process is an illustration of an example method. It should be understood that the specific order or hierarchy of steps in the process can be rearranged based on design preferences. Some steps can be performed simultaneously. The accompanying method claims present elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0086] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The foregoing description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but to conform to the full range consistent with the language claims, wherein a reference to an element in the singular is not intended to mean "one and only one", unless specifically so indicated, otherwise it means "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (such as his) include female and neutral genders (such as her and its), and vice versa. Titles and subtitles (if any) are only for convenience and do not limit the present invention.

[0087] As used herein, the term "website" may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, and the like. Therefore, the term "website" may be used interchangeably with the terms "web page" and "server." The predicate words "configured to," "operably," and "programmed" do not imply any specific tangible or intangible modification of the subject, but are intended to be used interchangeably. For example, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation, or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute the code, or operable to execute the code.

[0088] As used herein, the term "automatically" may include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0089] Phrases such as "aspects" do not mean that such aspects are essential to the subject technology or that such aspects apply to all configurations of the subject technology. Disclosures related to an aspect may apply to all configurations or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "embodiments" do not mean that such embodiments are essential to the subject technology or that such embodiments apply to all configurations of the subject technology. Disclosures related to embodiments may apply to all embodiments or one or more embodiments. An embodiment may provide one or more examples. Phrases such as "embodiments" may refer to one or more embodiments, and vice versa. Phrases such as "configurations" do not mean that such configurations are essential to the subject technology or that such configurations apply to all configurations of the subject technology. Disclosures about configurations may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configurations" may refer to one or more configurations, and vice versa.

[0090] All structural and functional equivalents of the elements throughout the various aspects described in the present disclosure that are known or will later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, all content disclosed herein, whether or not expressly claimed in the claims, is not intended to be freely contributed to the public. An element of a claim shall not be interpreted under the provisions of the sixth paragraph of 35 U.S.C. ξ 112 unless the element is expressly described by the term "means for..." or, in the case of a method claim, by the term "step for..." In addition, to the extent that the term "comprising," "having," and the like are used in the specification or claims, the term is intended to be inclusive, which is similar to the interpretation when "comprising" is used as a transitional word in a claim.

Claims

1. A computer-implemented method for recognizing electronic handwriting, comprising: receiving a new segment of electronic handwriting for display at a user-specified location within or adjacent to existing electronic content displayed on a touch-sensitive display device; generating a content action tree for the new segment, each lower node branching from a higher level node in the tree representing one of a plurality of content actions executable with respect to the existing electronic content to identify a portion of the existing electronic content to associate with at least a portion of the new segment, the identified portion of the existing electronic content being delineated according to a plurality of nodes along a given branch, automatically determining, without user intervention, candidate portions of the existing electronic content to provide to a handwriting recognition module along with the new segment based on automatically traversing the tree along respective paths from a root node of the tree to respective leaf nodes of the tree and identifying portions of the existing electronic content associated with content actions represented by nodes along the path; providing the received new segment together with each determined candidate portion of the existing electronic content to the handwriting recognition module to obtain a plurality of textual interpretations of the new segment; generating corresponding electronic handwriting representations for the plurality of text interpretations obtained from the handwriting recognition module; scoring the plurality of text interpretations obtained from the handwriting recognition module based on how each respective electronic handwriting representation would change the display of the existing electronic content if the respective electronic handwriting representation were displayed substantially at the user-specified location within or near the existing electronic content; selecting, based on the score, an electronic handwriting representation corresponding to a respective text interpretation of the plurality of text interpretations; as well as The existing electronic content is modified to include the selected electronic handwriting representation substantially located at the user-specified location for display on the display device.

2. The computer-implemented method of claim 1 , wherein: The electronic handwriting representation includes one or more computer-reproduced glyphs substantially in the handwriting style of the existing electronic content, and the existing electronic content is modified by dynamic re-formatting of at least a portion of the existing electronic content to accommodate the selected electronic handwriting representation.

3. The computer-implemented method of claim 1 , further comprising: determining that the score associated with the selected electronic handwriting representation does not satisfy a predetermined confidence level; as well as A portion of the modified electronic content corresponding to the selected electronic handwriting representation is visually identified as not satisfying the predetermined confidence level.

4. The computer-implemented method of claim 3, further comprising: Prior to modifying the existing electronic content, visually designating a computer-designated location within or near the existing electronic content that corresponds to an insertion point of the selected electronic ink; monitoring a user for an indication of acceptance or rejection of a selected electronic handwriting representation; as well as When an instruction of rejecting the selected electronic handwriting representation is received from the user, a second electronic handwriting representation corresponding to a second text interpretation among the plurality of text interpretations is selected as the selected electronic handwriting representation.

5. The computer-implemented method of claim 1 , further comprising: maintaining in memory a character-based text representation of the existing electronic content displayed on a display device; as well as The character-based text representation is updated to include the corresponding text interpretation corresponding to the selected electronic handwriting representation.

6. The computer-implemented method of claim 5, further comprising: An updated character-based text representation is provided for display on the display device along with the modified version of the existing electronic content, the updated character-based text representation comprising a plurality of words, each word being provided for display adjacent to or directly beneath a corresponding displayed word segment of the modified version of the existing electronic content.

7. The computer-implemented method of claim 1 , further comprising: receiving an indication that the selected electronic handwriting representation is rejected by the user; selecting a second electronic handwriting representation corresponding to a second text interpretation among the plurality of text interpretations as the selected electronic handwriting representation; as well as The existing electronic content is modified to include the second electronic handwriting representation substantially located at the user-specified location for display on the display device.

8. The computer-implemented method of claim 7, further comprising: rescoring the plurality of text interpretations obtained from the handwriting recognition module based on user input associated with an indication that the selected electronic handwriting representation is rejected and based on how each respective electronic handwriting representation would alter a display of the existing electronic content if the respective electronic handwriting representation were displayed substantially at the user-specified location within or adjacent to the existing electronic content; as well as Based on the re-scoring, the second electronic handwriting representation is selected.

9. The computer-implemented method of claim 1, wherein: The new segment of the electronic handwriting modifies the existing segment of the electronic handwriting of the existing electronic content, and wherein the existing segment of the electronic handwriting together with the new segment are provided to the handwriting recognition module to obtain the plurality of text interpretations.

10. The computer-implemented method of claim 1, wherein: The score is based on changes to at least one of: word spacing, baseline alignment, character kerning, character deformation, number of lines on the current page, word reflow, line reflow, or paragraph reflow.

11. A system for recognizing electronic handwriting, comprising: touch-sensitive display devices; one or more processors; A memory storage device having instructions stored on the memory storage device, the instructions, when executed by the one or more processors, configure a computing device to perform operations including: receiving, via the touch-sensitive display device, a new segment of electronic handwriting for display at a user-specified location within or adjacent to existing electronic content displayed on the touch-sensitive display device; generating a content action tree for the new segment, each lower node branching from a higher level node in the tree representing one of a plurality of content actions executable with respect to the existing electronic content to identify a portion of the existing electronic content to associate with at least a portion of the new segment, the identified portion of the existing electronic content being delineated according to a plurality of nodes along a given branch, automatically determining, without user intervention, candidate portions of the existing electronic content to provide to a handwriting recognition module along with the new segment based on automatically traversing the tree along respective paths from a root node of the tree to respective leaf nodes of the tree and identifying portions of the existing electronic content associated with content actions represented by nodes along the path; providing the received new segment together with each determined candidate portion of the existing electronic content to the handwriting recognition module to obtain a plurality of textual interpretations of the new segment; generating corresponding electronic handwriting representations for the plurality of text interpretations obtained from the handwriting recognition module; scoring the plurality of text interpretations obtained from the handwriting recognition module based on how each respective electronic handwriting representation would change the display of the existing electronic content if the respective electronic handwriting representation were displayed substantially at the user-specified location within or near the existing electronic content; selecting, based on the score, an electronic handwriting representation corresponding to a respective text interpretation of the plurality of text interpretations; as well as The existing electronic content is modified to include a selected electronic handwriting representation substantially located at the user-specified location for display on the touch-sensitive display device.

12. The system according to claim 11, wherein: The operations also include: determining that the score associated with the selected electronic handwriting representation does not satisfy a predetermined confidence level; and A portion of the modified electronic content corresponding to the selected electronic handwriting representation is visually identified as not satisfying the predetermined confidence level.

13. The system of claim 12, further comprising: Prior to modifying the existing electronic content, visually designating a computer-designated location within or near the existing electronic content that corresponds to an insertion point of the selected electronic ink; monitoring a user for an indication of acceptance or rejection of a selected electronic handwriting representation; as well as When an instruction of rejecting the selected electronic handwriting representation is received from the user, a second electronic handwriting representation corresponding to a second text interpretation among the plurality of text interpretations is selected as the selected electronic handwriting representation.

14. The system of claim 11, further comprising: maintaining in memory a character-based text representation of the existing electronic content displayed on the touch-sensitive display device; as well as The character-based text representation is updated to include the corresponding text interpretation corresponding to the selected electronic handwriting representation.

15. The system of claim 14, further comprising: An updated character-based text representation is provided for display on the touch-sensitive display device along with the modified version of the existing electronic content, the updated character-based text representation comprising a plurality of words, each word being provided for display adjacent to or directly beneath a corresponding displayed word segment of the modified version of the existing electronic content.

16. The system of claim 11, further comprising: receiving an indication that the selected electronic handwriting representation is rejected by the user; selecting a second electronic handwriting representation corresponding to a second text interpretation among the plurality of text interpretations as the selected electronic handwriting representation; as well as The existing electronic content is modified to include the second electronic handwriting representation located substantially at the user-specified location for display on the touch-sensitive display device.

17. The system of claim 16, further comprising: rescoring the plurality of text interpretations obtained from the handwriting recognition module based on user input associated with an indication that the selected electronic handwriting representation is rejected and based on how each respective electronic handwriting representation would alter a display of the existing electronic content if the respective electronic handwriting representation were displayed substantially at the user-specified location within or adjacent to the existing electronic content; as well as Based on the re-scoring, the second electronic handwriting representation is selected.

18. The system of claim 11, wherein: The new segment of the electronic handwriting modifies the existing segment of the electronic handwriting of the existing electronic content, and wherein the existing segment of the electronic handwriting together with the new segment are provided to the handwriting recognition module to obtain the plurality of text interpretations.

19. The system of claim 11, wherein: The score is based on changes to at least one of: word spacing, baseline alignment, character kerning, character deformation, number of lines on the current page, word reflow, line reflow, or paragraph reflow.

20. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, configure the computing device to perform operations comprising: receiving, via a touch-sensitive display device, a new segment of electronic handwriting for display at a user-specified location within or adjacent to existing electronic content displayed on the touch-sensitive display device; generating a content action tree for the new segment, each lower node branching from a higher level node in the tree representing one of a plurality of content actions executable with respect to the existing electronic content to identify a portion of the existing electronic content to associate with at least a portion of the new segment, the identified portion of the existing electronic content being delineated according to a plurality of nodes along a given branch, automatically determining, without user intervention, candidate portions of the existing electronic content to provide to a handwriting recognition module along with the new segment based on automatically traversing the tree along respective paths from a root node of the tree to respective leaf nodes of the tree and identifying portions of the existing electronic content associated with content actions represented by nodes along the path; providing the received new segment together with each determined candidate portion of the existing electronic content to the handwriting recognition module to obtain a plurality of textual interpretations of the new segment; generating corresponding electronic handwriting representations for the plurality of text interpretations obtained from the handwriting recognition module; scoring the plurality of text interpretations obtained from the handwriting recognition module based on how each respective electronic handwriting representation would change the display of the existing electronic content if the respective electronic handwriting representation were displayed substantially at the user-specified location within or near the existing electronic content; selecting, based on the score, an electronic handwriting representation corresponding to a respective text interpretation of the plurality of text interpretations; as well as The existing electronic content is modified to include a selected electronic handwriting representation substantially located at the user-specified location for display on the touch-sensitive display device.

Images