Management for real-time handwriting recognition

Abstract

To provide a functionality of handwriting input in a user device.SOLUTION: A handwriting recognition module is trained to have a repertory including a plurality of non-duplicate scripts, and can use a single handwriting recognition model to recognize tens of thousands of characters. A handwriting input module provides real-time stroke order and stroke direction independent handwriting recognition relating to handwriting input for plural characters. Specifically, the real-time stroke order and stroke direction independent handwriting recognition is provided relating to Chinese handwriting recognition at a level of plural characters or sentences. A user interface for providing a functionality of the handwriting input is also disclosed.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] This specification relates to providing handwriting input functionality on a computing device, and more particularly to providing real-time, multi-script, stroke-order independent handwriting recognition and input functionality on a computing device. [Background technology]

[0002] Handwriting input is an important alternative input method for computing devices equipped with touch-sensitive surfaces (e.g., touch-sensitive display screens or touchpads). Many users, particularly those in some Asian or Arab countries, are accustomed to writing in cursive and may feel comfortable writing in regular handwriting as opposed to typing on a keyboard.

[0003] For certain logographic writing systems, such as Chinese Hanzi and Japanese Kanji (also referred to as Sinophone), alternative syllabic input methods (e.g., pinyin or kana) are available for inputting the corresponding logographic writing system characters, but such syllabic input methods are inappropriate if the user does not know how to phonetically spell the logographic character and if they use an incorrect phonetic spelling of the logographic character. Therefore, the ability to use handwriting input on a computing device is crucial for users who are unable to pronounce words well or at all for the associated logographic writing system.

[0004] Although handwriting input functionality has become somewhat widespread in certain regions of the world, there is still a need for improvement. Specifically, human handwriting is highly variable (e.g., in terms of stroke order, size, typeface, etc.), and high-quality handwriting recognition software is complex and requires extensive training. Therefore, providing efficient real-time handwriting recognition on mobile devices with limited memory and computing resources is a challenging task.

[0005] Furthermore, in today's multicultural world, users in many countries use multiple languages ​​and may frequently need to write in more than one script (e.g., mentioning a movie title in English, writing a message in Chinese). However, manually switching the recognition system to the desired script or language while writing is cumbersome and inefficient. Furthermore, the usefulness of conventional multi-script handwriting recognition techniques is severely limited because expanding a device's recognition capabilities to handle multiple scripts simultaneously significantly increases the complexity of the recognition system and the demands on computer resources.

[0006] Furthermore, conventional handwriting techniques rely heavily on language- or script-specific idiosyncrasies to achieve recognition accuracy. Such idiosyncrasies are not easily portable to other languages ​​or scripts. Therefore, adding handwriting input capabilities for new languages ​​or scripts is a difficult task that is not easily undertaken by software and device suppliers. As a result, multilingual users are denied significant alternative input methods for their electronic devices.

[0007] Conventional user interfaces for providing handwriting input include an area for accepting handwriting input from a user and an area for displaying handwriting recognition results. Significant improvements in user interfaces are still needed to improve efficiency, accuracy, and the overall user experience on portable devices with small form factors. DISCLOSURE OF THE INVENTION

[0008] This specification describes techniques for providing multi-script handwriting recognition using a generic recognizer. The generic recognizer is trained using a large, multi-script corpus of writing samples of characters in various languages ​​or scripts. The training of the generic recognizer is language-independent, script-independent, stroke-order-independent, and stroke-direction-independent. Therefore, the same recognizer can recognize mixed-language, mixed-script handwriting input without requiring manual switching between input languages ​​during use. Furthermore, the generic recognizer is lightweight enough to be deployed as a stand-alone module on a mobile device, thereby enabling handwriting input in various languages ​​and scripts used in various regions around the world.

[0009] Moreover, because the generalized recognizer is trained on stroke-order-independent and stroke-direction-independent spatially derived features and does not require stroke-level temporal or sequence information, the generalized recognizer offers numerous additional features and advantages over traditional time-based recognition methods (e.g., Hidden Markov Model (HMM)-based recognition methods). For example, a user can input one or more character, word, and sentence strokes in any order and still obtain the same recognition results. Therefore, out-of-order multiple-character input and out-of-order modification (e.g., addition or rewriting) of previously input characters are possible.

[0010] Furthermore, the generic recognizer may also be used in connection with real-time handwriting recognition, where temporal information about each stroke is available and optionally used to disambiguate or segment the handwritten input before character recognition is performed by the generic recognizer. The real-time, stroke-order-independent recognition described herein differs from and may provide better performance than traditional offline recognition methods (e.g., optical character recognition (OCR)). Furthermore, the generic recognizer described herein is able to handle high variability in individual writing habits (e.g., variability in speed, sloshing, stroke order, stroke direction, stroke continuity, etc.) without explicitly incorporating salient features of these variations into the recognition system, thereby reducing the overall complexity of the recognition system.

[0011] As described herein, in some embodiments, temporally derived stroke distribution information is optionally reintroduced into the generic recognizer to improve recognition accuracy and remove ambiguity between similar-looking recognition outputs for the same input image. The reintroduction of temporally derived stroke distribution information does not compromise the stroke order and stroke direction independence of the generic recognizer, because the temporally derived features and the spatially derived features are obtained through separate training processes and are combined in the handwriting recognition model only after the separate training processes are completed. Furthermore, the temporally derived stroke distribution information is carefully designed to capture salient temporal characteristics of similar-looking characters without relying on explicit knowledge of stroke order differences between similar-looking characters.

[0012] A user interface for providing handwriting input functionality is also described herein.

[0013] In some embodiments, a method for providing multi-script handwriting recognition includes training a multi-script handwriting recognition model based on spatially derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts, and providing real-time handwriting recognition of a user's handwritten input using the multi-script handwriting recognition model that has been trained on the spatially derived features of the multi-script training corpus.

[0014] In some embodiments, a method for providing multi-script handwriting recognition includes receiving a multi-script handwriting recognition model, the multi-script recognition model being trained on spatially derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; receiving handwritten input from a user, the handwritten input including one or more handwritten strokes provided on a touch-sensitive surface coupled to a user device; and in response to receiving the handwritten input, providing one or more handwriting recognition results to the user in real time based on the multi-script handwriting recognition model that has been trained on spatially derived features of the multi-script training corpus.

[0015] In some embodiments, a method for providing real-time handwriting recognition includes receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to a handwritten character; generating an input image based on the plurality of handwritten strokes; providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten character, the handwriting recognition model providing stroke-order independent handwriting recognition; and displaying a same first output character in real time upon receiving the plurality of handwritten strokes regardless of the respective order of the plurality of handwritten strokes received from the user.

[0016] In some embodiments, the method further includes receiving a second plurality of handwritten strokes from a user, the second plurality of handwritten strokes corresponding to a second handwritten character; generating a second input image based on the second plurality of handwritten strokes; providing the second input image to a handwriting recognition model to perform real-time recognition of the second handwritten character; and displaying a second output character corresponding to the second plurality of handwritten strokes in real time of receiving the second plurality of handwritten strokes, wherein the first output character and the second output character are displayed simultaneously in a spatial arrangement independent of the respective order of the first plurality of handwritten inputs and the second plurality of handwritten inputs provided by the user.

[0017] In some embodiments, the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, and the second output character follows the first output character in spatial arrangement along the default writing direction, the method further comprising: receiving third handwritten strokes from a user to correct the handwritten character, the third handwritten strokes being received later in time than the first plurality of handwritten strokes and the second plurality of handwritten strokes; and, in response to receiving the third handwritten strokes, correcting the third handwritten strokes by adjusting the third handwritten strokes relative to the first plurality of handwritten strokes. The method further includes assigning the handwritten strokes to the same recognition unit as the first plurality of handwritten strokes based on relative proximity to the strokes; generating a corrected input image based on the first plurality of handwritten strokes and the third handwritten stroke; providing the corrected input image to a handwriting recognition model to perform real-time recognition of the corrected handwritten character; and, in response to receiving the third handwritten input, displaying a third output character corresponding to the corrected input image, wherein the third output character replaces the first output character and is displayed simultaneously with the second output character in a spatial arrangement along the default writing direction.

[0018] In some embodiments, the method further includes receiving a delete input from a user while the third output character and the second output character are simultaneously displayed as recognition results in a candidate display area of ​​the handwriting input interface, and in response to the delete input, deleting the second output character from the recognition result while maintaining the third output character in the recognition result.

[0019] In some embodiments, rendering a first plurality of handwritten strokes, a second plurality of handwritten strokes, and a third plurality of handwritten strokes in real time within a handwriting input area of ​​a handwriting input interface as each of the handwritten strokes is provided by a user, and in response to receiving a delete input, deleting the respective renderings of the second plurality of handwritten strokes from the handwriting input area while maintaining the respective renderings of the first plurality of handwritten strokes and the third handwritten strokes within the handwriting input area.

[0020] In some embodiments, a method for providing real-time handwriting recognition includes receiving handwritten input from a user, the handwritten input including one or more handwritten strokes provided in a handwriting input area of ​​a handwriting input interface; identifying a plurality of output characters for the handwritten input based on a handwriting recognition model; classifying the plurality of output characters into two or more categories based on predetermined categorization criteria; displaying each output character in a first category of the two or more categories in an initial view of a candidate display area of ​​the handwriting input interface, the initial view of the candidate display area being provided simultaneously with an affordance for invoking an expanded view of the candidate display area; receiving user input selecting the affordance for invoking the expanded view; and displaying, in response to the user input, each output character in the first category and each output character in at least a second category of the two or more categories that were not previously displayed in the initial view of the candidate display area.

[0021] In some embodiments, a method for providing real-time handwriting recognition includes receiving handwritten input from a user, the handwritten input including a plurality of handwritten strokes provided in a handwriting input area of ​​a handwriting input interface; recognizing a plurality of output characters from the handwritten input based on a handwriting recognition model, the output characters including at least a first emoji and at least a first character from a script of a natural human language; and displaying recognition results including the first emoji and the first character from the script of the natural human language in a candidate display area of ​​the handwriting input interface.

[0022] In some embodiments, a method of providing handwriting recognition includes receiving handwritten input from a user, the handwritten input including a plurality of handwritten strokes provided in a touch-sensitive surface coupled to a device; rendering the plurality of handwritten strokes in real time within a handwriting input area of ​​a handwriting input interface; receiving one of a pinch gesture input and a magnification gesture input over the plurality of handwritten strokes; upon receiving the pinch gesture input, generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit; upon receiving the magnification gesture input, generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the magnification gesture input; and upon generating each of the first recognition result and the second recognition result, displaying the generated recognition result in a candidate display area of ​​the handwriting input interface.

[0023] In some embodiments, a method for providing handwriting recognition includes receiving handwritten input from a user, the handwritten input including a plurality of handwritten strokes provided in a handwriting input area of ​​a handwriting input interface; identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a corresponding subset of the plurality of handwritten strokes; generating a multi-character recognition result including respective characters recognized from the plurality of recognition units; displaying the multi-character recognition result in a candidate display area of ​​the handwriting input interface; receiving a delete input from the user while the multi-character recognition result is displayed in the candidate display area; and removing a last character from the multi-character recognition result displayed in the candidate display area in response to receiving the delete input.

[0024] In some embodiments, a method for providing real-time handwriting recognition includes determining an orientation of a device; and, according to the device being in a first orientation, providing a handwriting input interface on the device in a horizontal input mode, wherein each line of handwriting input entered in the horizontal input mode is divided into one or more respective recognition units along a horizontal writing direction; and, according to the device being in a second orientation, providing a handwriting input interface on the device in a vertical input mode, wherein each line of handwriting input entered in the vertical input mode is divided into one or more respective recognition units along the vertical writing direction.

[0025] In some embodiments, a method for providing real-time handwriting recognition includes receiving handwritten input from a user, the handwritten input including a plurality of handwritten strokes provided on a touch-sensitive surface coupled to a device; rendering the plurality of handwritten strokes within a handwriting input area of ​​a handwriting input interface; dividing the plurality of handwritten strokes into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes; receiving an edit request from the user; visually identifying the two or more recognition units within the handwriting input area in response to the edit request; and providing means for individually deleting each of the two or more recognition units from the handwriting input area.

[0026] In some embodiments, a method for providing real-time handwriting recognition includes receiving a first handwritten input from a user, the first handwritten input including a plurality of handwritten strokes forming a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of ​​a handwriting input interface; rendering each of the plurality of handwritten strokes in the handwriting input area as the handwritten strokes are provided by the user; initiating a respective fading process for each of the plurality of recognition units after the recognition unit is fully rendered, during which the rendering of the recognition unit in the first handwritten input gradually fades; receiving a second handwritten input from the user over an area of ​​the handwriting input area that was occupied by the faded recognition unit of the plurality of recognition units; and, in response to receiving the second handwritten input, rendering the second handwritten input in the handwriting input area; and erasing all faded recognition units from the handwriting input area.

[0027] In some embodiments, a method for providing handwriting recognition includes separately training a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, where the set of spatially derived features is trained on a corpus of training images, each of which is an image of a handwriting sample for a respective character of an output character set, and the set of temporally derived features is trained on a corpus of stroke distribution profiles, each stroke distribution profile numerically characterizing the spatial distribution of a plurality of strokes within the handwriting sample for a respective character of the output character set; separately training a handwriting recognition model that combines the set of spatially derived features and the set of temporally derived features; and using the handwriting recognition model to provide real-time handwriting recognition of a user's handwritten input.

[0028] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

[0030] [Figure 2] 1 illustrates a portable multifunction device having a touch-sensitive display in accordance with some embodiments.

[0031] [Figure 3] 1 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments. FIG.

[0032] [Figure 4]1 illustrates an exemplary user interface for a multifunction device with a touch-sensitive surface that is separate from the display in accordance with some embodiments.

[0033] [Figure 5] FIG. 1 is a block diagram of an operating environment for a handwriting input system, according to some implementations.

[0034] [Figure 6] FIG. 1 is a block diagram of a multi-script handwriting recognition model according to some embodiments.

[0035] [Figure 7] 1 is a flow diagram of an exemplary process for training a multi-script handwriting recognition model according to some embodiments.

[0036] [Figure 8A] 1 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device according to some embodiments. [Figure 8B] 1 illustrates an exemplary user interface showing real-time multi-script handwriting recognition and input on a portable multifunction device according to some embodiments.

[0037] [Figure 9A] 1 is a flow diagram of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device. [Figure 9B] 1 is a flow diagram of an exemplary process for providing real-time multi-script handwriting recognition and input on a portable multifunction device.

[0038] [Figure 10A] 1 is a flow diagram of an exemplary process for providing real-time, stroke-order-independent handwriting recognition and input on a portable multifunction device according to some embodiments. [Figure 10B] 1 is a flow diagram of an exemplary process for providing real-time, stroke-order-independent handwriting recognition and input on a portable multifunction device according to some embodiments. [Figure 10C] 1 is a flow diagram of an exemplary process for providing real-time, stroke-order-independent handwriting recognition and input on a portable multifunction device according to some embodiments.

[0039] [Figure 11A] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11B] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11C] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11D] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11E] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11F]1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11G] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11H] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11I] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11J] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments. [Figure 11K] 1 illustrates an exemplary user interface for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, in accordance with some embodiments.

[0040] [Figure 12A] 1 is a flowchart of an exemplary process for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, according to some embodiments. [Figure 12B]1 is a flowchart of an exemplary process for selectively displaying one category of recognition results in a normal view of the candidate display area and another category of recognition results in an expanded view of the candidate display area, according to some embodiments.

[0041] [Figure 13A] 1 illustrates an exemplary user interface for inputting emojis through handwriting input according to some embodiments. [Figure 13B] 1 illustrates an exemplary user interface for inputting emojis through handwriting input according to some embodiments. [Figure 13C] 1 illustrates an exemplary user interface for inputting emojis through handwriting input according to some embodiments. [Figure 13D] 1 illustrates an exemplary user interface for inputting emojis through handwriting input according to some embodiments. [Figure 13E] 1 illustrates an exemplary user interface for inputting emojis through handwriting input according to some embodiments.

[0042] [Figure 14] 1 is a flow diagram of an exemplary process for inputting emojis through handwriting input according to some embodiments.

[0043] [Figure 15A] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15B] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15C]1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15D] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15E] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15F] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15G] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15H] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15I] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15J]1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments. [Figure 15K] 1 illustrates an exemplary user interface for informing a handwriting input module how to separate currently stored handwriting input into one or more recognition units using a pinch or spread gesture, according to some embodiments.

[0044] [Figure 16A] 1 is a flow diagram of an exemplary process for using a pinch or spread gesture to inform a handwriting input module how to separate currently stored handwriting input into one or more recognition units, according to some embodiments. [Figure 16B] 1 is a flow diagram of an exemplary process for using a pinch or spread gesture to inform a handwriting input module how to separate currently stored handwriting input into one or more recognition units, according to some embodiments.

[0045] [Figure 17A] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17B] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17C] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17D] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17E]1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17F] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17G] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 17H] 1 illustrates an exemplary user interface for providing character-by-character deletion of a user's handwritten input according to some embodiments.

[0046] [Figure 18A] 1 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwritten input according to some embodiments. [Figure 18B] 1 is a flow diagram of an exemplary process for providing character-by-character deletion of a user's handwritten input according to some embodiments.

[0047] [Figure 19A] 10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments. [Figure 19B] 10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments. [Figure 19C] 10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments. [Figure 19D] 10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments. [Figure 19E] 10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments. [Figure 19F]10 illustrates an exemplary user interface for switching between vertical and horizontal writing modes according to some embodiments.

[0048] [Figure 20A] 1 is a flow diagram of an exemplary process for switching between vertical and horizontal writing modes according to some embodiments. [Figure 20B] 1 is a flow diagram of an exemplary process for switching between vertical and horizontal writing modes according to some embodiments. [Figure 20C] 1 is a flow diagram of an exemplary process for switching between vertical and horizontal writing modes according to some embodiments.

[0049] [Figure 21A] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21B] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21C] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21D] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21E] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21F] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21G]1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 21H] 1 illustrates a user interface for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments.

[0050] [Figure 22A] 1 is a flow diagram of an exemplary process for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments. [Figure 22B] 1 is a flow diagram of an exemplary process for providing a means for viewing and selectively deleting individual recognition units identified within a user's handwritten input, according to some embodiments.

[0051] [Figure 23A] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23B] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23C] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23D]1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23E] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23F] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23G] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23H] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23I] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23J] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23K]1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 23L] 1 illustrates an exemplary user interface for utilizing new handwritten input provided on top of an existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments.

[0052] [Figure 24A] FIG. 10 is a flow diagram of an exemplary process for utilizing new handwritten input provided over existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments. [Figure 24B] FIG. 10 is a flow diagram of an exemplary process for utilizing new handwritten input provided over existing handwritten input in a handwritten input area as an implicit confirmation input for entering a displayed recognition result for the existing handwritten input, according to some embodiments.

[0053] [Figure 25A] FIG. 10 is a flow diagram of an exemplary process for integrating temporally derived stroke distribution information into a handwriting recognition model based on spatially derived features without compromising the stroke order and stroke direction dependency of the handwriting recognition model, according to some embodiments; [Figure 25B] FIG. 10 is a flow diagram of an exemplary process for integrating temporally derived stroke distribution information into a handwriting recognition model based on spatially derived features without compromising the stroke order and stroke direction dependency of the handwriting recognition model, according to some embodiments;

[0054] [Figure 26]FIG. 1 is a block diagram illustrating separate training and subsequent integration of spatially and temporally derived features of an exemplary handwriting recognition system according to some embodiments;

[0055] [Figure 27] FIG. 2 is a block diagram illustrating an exemplary method for calculating a stroke distribution profile of a character.

[0056] Like reference numbers refer to corresponding parts throughout these drawings. DETAILED DESCRIPTION OF THE INVENTION

[0057] Many electronic devices have a graphical user interface with a soft keyboard for character input. On some electronic devices, a user may also be able to install or enable a handwriting input interface that allows the user to input characters via handwriting on a touch-sensitive display screen or touch-sensitive surface coupled to the device. Conventional handwriting recognition input methods and user interfaces have several problems and drawbacks. For example, Typically, traditional handwriting input functionality is enabled on a per-language or per-script basis. Each added input language requires the installation of a separate handwriting recognition model, which consumes separate storage space and memory. The synergistic effects achieved by combining handwriting recognition models for different languages ​​are minimal, and mixed-language or mixed-script handwriting recognition has traditionally taken a very long time due to complex disambiguation processes. Furthermore, conventional handwriting recognition systems rely heavily on language-specific or script-specific characteristics for character recognition. Recognition of mixed-language handwriting input has low accuracy. Furthermore, the available combinations of recognized languages ​​are very limited. Most systems require the user to manually specify the desired language-specific handwriting recognizer before providing handwriting input in each non-default language or script. Many existing real-time handwriting recognition models require temporal or alignment information at the stroke-by-stroke level, which leads to inaccurate recognition results when dealing with the high variability in how characters can be written (e.g., high variability in stroke shape, length, velocity, division, order, and direction due to writing style or personal habits). Some systems also require users to adhere to strict spatial and temporal criteria (e.g., with built-in assumptions about the size, alignment, and time frame of each character input) when providing handwritten input. Any deviation from these criteria has led to inaccurate recognition results that are difficult to correct. Currently, most real-time handwriting input interfaces only allow users to input a few characters at a time. Long words or sentences are broken down into shorter segments and input separately. This formal input not only places a cognitive burden on the user to maintain compositional flow, but also makes it difficult for the user to modify or correct previously entered characters or phrases.

[0058] The embodiments described below address these and related issues.

[0059] Figures 1-4 below provide descriptions of exemplary devices. Figures 5, 6, and 26-27 show exemplary handwriting recognition and input systems. Figures 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, and 23A-12L show exemplary user interfaces for handwriting recognition and input. Figures 7, 9A, 9B, 10A-10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A-20C, 22A, 22B, 24A, 24B, and 25 are flow diagrams illustrating methods for enabling handwriting recognition and input, including training a handwriting recognition model, providing real-time handwriting recognition results, providing means for inputting and correcting handwritten input, and providing means for inputting the recognition results as text input. The user interfaces in Figures 8A, 8B, 11A-11K, 13A-13E, 15A-15K, 17A-17F, 19A-19F, 21A-21H, and 23A-12L are used to illustrate the processes in Figures 7, 9A, 9B, 10A-10C, 12A, 12B, 14, 16A, 16B, 18A, 18B, 20A-20C, 22A, 22B, 24A, 24B, and 25. Exemplary Devices

[0060] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail as not to unnecessarily obscure aspects of the embodiments.

[0061] It will also be understood that although terms such as "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the present invention. Although a first contact and a second contact are both contacts, they are not the same contact.

[0062] The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "and / or" will also be understood to refer to and encompass any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0063] As used herein, the term "in the event of" can be interpreted to mean "when" or "when," or "in response to a determination that" or "in response to the detection of," depending on the context. Similarly, the phrase "when it is determined that" or "when (a described condition or event) is detected" can be interpreted to mean "when it is determined that" or "in response to a determination that," or "when (a described condition or event) is detected" or "in response to the detection of (a described condition or event)," depending on the context.

[0064] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices by Apple Inc. (Cupertino, California). Other portable electronic devices, such as laptop or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads), can also be used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touchscreen displays and / or touchpads).

[0065] The following discussion describes an electronic device that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0066] The device typically supports a variety of applications, such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disc authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0067] Various applications that may run on the device may use at least one common physical user interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed on the device may be adjusted and / or changed for each application and / or within each application. In this manner, the device's common physical architecture (such as the touch-sensitive surface) may support various applications using a user interface that is intuitive and transparent to the user.

[0068] Attention now turns to embodiments of portable devices with touch-sensitive displays. FIG. 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display 112, according to some embodiments. Touch-sensitive display 112 may conveniently be referred to as a "touch screen" and may also be known or referred to as a touch-sensitive display system. Device 100 may include memory 102 (which may include one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripherals interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 may include one or more light sensors 164. These components may communicate via one or more communication buses or signal lines 103.

[0069] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 may have more or fewer components than those shown, may combine two or more components, or may have a different configuration or arrangement of its components. The various components shown in Figure 1 may be implemented as hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0070] Memory 102 may include high-speed random-access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and peripherals interface 118, may be controlled by memory controller 122.

[0071] A peripheral interface 118 may be used to couple input and output peripherals of the device to the CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions and process data for the device 100.

[0072] In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. In some other embodiments, they may be implemented on separate chips.

[0073] RF (radio frequency) circuitry 108 transmits and receives RF signals, also called electromagnetic signals, that convert electrical signals to and from electromagnetic signals and communicate with communication networks and other communication devices via the electromagnetic signals.

[0074] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripherals interface 118 for processing. The audio data can be retrieved from and / or transmitted to memory 102 and / or RF circuitry 108 by peripherals interface 118. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., 212 in FIG. 2 ).

[0075] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touchscreen 112 and other input control devices 116, to a peripheral interface 118. The I / O subsystem 106 may include a display controller 156 and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals to / from the other input or control devices 116. The other input control devices 116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 may be coupled to any (or none) of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in FIG. 2) may include up / down buttons for adjusting the volume of the speaker 111 and / or microphone 113. The one or more buttons may include a push button (e.g., 206 in FIG. 2).

[0076] Touch-sensitive display 112 provides an input and output interface between the device and a user. Display controller 156 receives and / or sends electrical signals to and from touchscreen 112. Touchscreen 112 displays visual output to the user. This visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of these visual outputs may correspond to user interface objects.

[0077] Touchscreen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction sets in memory 102) detects contacts (and any movement or disruption of that contact) on touchscreen 112 and translates the detected contacts into interactions with user interface objects (e.g., one or more softkeys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the point of contact between touchscreen 112 and the user corresponds to the user's finger.

[0078] Touchscreen 112 may use LCD (liquid crystal display), LPD (light emitting polymer display), or LED (light emitting diode) technology, although other display technologies may be used in other embodiments. Touchscreen 112 and display controller 156 may detect contact and any movement or disruption of that contact using any of a number of now known or later developed touch sensing technologies, including, but not limited to, capacitive, resistive, infrared, and surface ultrasonic technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® by Apple Inc. (Cupertino, California).

[0079] The touchscreen 112 may have a video resolution greater than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. A user can contact the touchscreen 112 using any suitable object or accessory, such as a stylus, a finger, or the like. In some embodiments, the user interface is designed to function primarily with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device converts coarse finger-based input into precise pointer / cursor positions or commands for performing actions desired by the user. Handwriting input can be provided on the touchscreen 112 via the location and movement of finger-based or stylus-based contacts. In some embodiments, the touchscreen 112 renders finger-based or stylus-based input as immediate visual feedback for the current handwriting input, providing the visual effect of actually writing on a writing surface (e.g., a sheet of paper) using a writing instrument (e.g., a pen).

[0080] In some embodiments, in addition to the touchscreen, device 100 may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad can be a touch-sensitive surface separate from touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0081] Device 100 also includes a power system 162 for providing power to the various components. Power system 162 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with generating, managing, and distributing electrical power within a portable device.

[0082] Device 100 may also include one or more light sensors 164. FIG. 1 shows a light sensor coupled to light sensor controller 158 in I / O subsystem 106. Light sensor 164 may include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Light sensor 164 receives light from the environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), light sensor 164 can capture still images or video.

[0083] Device 100 may also include one or more proximity sensors 166. Figure 1 shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 may be coupled to input controller 160 in I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touchscreen 112 when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call).

[0084] Device 100 may also include one or more accelerometers 168. FIG. 1 shows accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 may be coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on analysis of data received from the one or more accelerometers. Device 100 optionally includes, in addition to accelerometer 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the location and orientation (e.g., portrait or landscape) of device 100.

[0085] In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Additionally, in some embodiments, as shown in FIGS. 1 and 3, memory 102 stores a handwriting input module 157. Handwriting input module 157 includes handwriting recognition models and provides handwriting recognition and input functionality to a user of device 100 (or device 300). Further details of handwriting input module 157 are provided in connection with FIGS. 5-27 and their accompanying descriptions.

[0086] Operating system 126 (e.g., an embedded operating system such as Darwin®, RTXC®, LINUX®, UNIX®, OS X®, WINDOWS®, or VxWorks®) includes various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0087] Communications module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by RF circuitry 108 and / or external ports 124. External ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) are adapted to couple directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.).

[0088] Contact / motion module 130 (in conjunction with display controller 156) can detect contact with touch screen 112 and with other touch-sensing devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes various software components for performing various operations related to detecting contact, such as determining whether contact occurs (e.g., detecting a finger-down event), determining whether there is contact movement and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-drag events), and determining whether contact has stopped (e.g., detecting a finger-up event or contact cessation). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact, as represented by a sequence of contact data, may include determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact. These operations can apply to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multitouch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on a touchpad.

[0089] Contact / motion module 130 can detect gesture input by a user. Different gestures on a touch-sensitive surface have different contact patterns. Therefore, a gesture can be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event, followed by a finger lift (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by one or more finger drag events, followed by a finger lift (lift-off) event.

[0090] Contact / motion module 130 is optionally utilized by handwriting input module 157 to register the input of handwritten strokes within a handwriting input area of ​​a handwriting input interface displayed on touch-sensitive display screen 112 (or within an area of ​​touchpad 355 corresponding to the handwriting input area displayed on display 340 in FIG. 3 ). In some embodiments, the location, movement path, and intensity associated with contacts during an initial finger-down event, a final finger-up event, and any time in between are recorded as handwritten strokes. Based on such information, handwritten strokes can be rendered on the display as feedback regarding the user input. Furthermore, one or more input images can be generated based on the handwritten strokes registered by contact / motion module 130.

[0091] Graphics module 132 includes various known software components for rendering and displaying graphics on touchscreen 112 or other display, including components for changing the brightness of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, video, animation, etc.

[0092] In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic may be assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, along with coordinate data and other graphic characteristic data, as appropriate, from an application or the like, and then generates screen image data for output to display controller 156.

[0093] Text input module 134, which may be a component of graphics module 132, provides a soft keyboard for entering text within various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input). In some embodiments, handwriting input module 157 is optionally invoked through the user interface of text input module 134, for example, through a keyboard selection affordance. In some embodiments, the same or a similar keyboard selection affordance is also provided within the handwriting input interface for invoking text input module 134.

[0094] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing, to the camera 143 as photo / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0095] The application 136 may include the following modules (or sets of instructions), or a subset or superset thereof: a contacts module 137 (sometimes called an address book or contact list); a phone module 138; a video conferencing module 139; an email client module 140; an instant messaging (IM) module 141; a training support module 142; a camera module 143 for still and / or video images; an image management module 144; a browser module 147; a calendar module 148; a weather widget 149; a stock price widget 150; a weather widget 151; a stock price widget 152; a weather widget 153; a stock price widget 154; a weather widget 155; a stock price widget 156; a weather widget 157; a stock price widget 158; a weather widget 159; a stock price widget 160; a weather widget 161; a stock price widget 162; a stock price widget 163; a stock price widget 164; a stock price widget 165; a stock price widget 166; a stock price widget 167; a stock price widget 168; a stock price widget 169; a stock price widget 170; a stock price widget 171; a stock price widget 172; a stock price widget 173; a stock price widget 174; a stock price widget 175; a stock price widget 176; a stock price widget 177; a stock price widget 178; a stock price widget 179; a stock price widget 180; a stock price widget 181; a stock price widget 182; a stock price widget 183; a stock price widget 184; a stock price widget 185; a stock price widget 186; a stock price widget 187; a stock price widget 188; a stock price widget 189; a stock price widget 190; a stock price widget 191 widget module 149, which may include one or more of widgets such as widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, as well as user-created widgets 149-6; widget creation module 150 for creating user-created widgets 149-6; search module 151; video and music player module 152, which may consist of a video player module and a music player module; notes module 153; map module 154; and / or online video module 155.

[0096] Examples of other applications 136 that may be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0097] In conjunction with touch screen 112, display controller 156, touch module 130, graphics module 132, handwriting input module 157, and text input module 134, contacts module 137 can be used to manage an address book or contact list (e.g., stored in memory 102 or in the application internal state 192 of contacts module 137 in memory 370), including adding names to the address book, removing names from the address book, associating phone numbers, email addresses, physical addresses, or other information with names, associating images with names, categorizing and sorting names, providing phone numbers or email addresses to initiate and / or facilitate communication, such as by telephone 138, video conference 139, email 140, or IM 141, and the like.

[0098] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, telephone module 138 can be used to enter a series of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify an entered telephone number, dial each telephone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As noted above, this wireless communication can use any of a number of communication standards, protocols, and technologies.

[0099] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact module 130, graphics module 132, handwriting input module 157, text input module 134, contact list 137, and telephone module 138, videoconferencing module 139 includes executable instructions for initiating, conducting, and terminating a videoconference between a user and one or more other participants in accordance with the user's instructions.

[0100] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, email client module 140 contains executable instructions for composing, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 greatly facilitates composing and sending emails with still or video images captured by camera module 143.

[0101] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, instant messaging module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages may include graphics, photos, audio files, video files, and / or other attachments, such as those supported by MMS and / or Extended Messaging Service (EMS). As used herein, "instant messaging" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0102] In conjunction with the RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, map module 154, and music player module 146, the training support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals), communicating with training sensors (sports devices), receiving training sensor data, calibrating sensors used to monitor workouts, selecting and playing music for workouts, and displaying, storing, and transmitting workout data.

[0103] In conjunction with touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or video (including video streams), storing them in memory 102, modifying characteristics of the still images or video, or deleting the still images or video from memory 102.

[0104] In conjunction with touchscreen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and / or video images.

[0105] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, browser module 147 contains executable instructions for browsing the Internet in accordance with a user's instructions, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0106] In association with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) according to user instructions.

[0107] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget modules 149 are mini-applications that can be downloaded and used by a user (e.g., weather widget 149-1, stock quotes widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or created by a user (e.g., user-created widget 149-6). In some embodiments, widgets include HTML (HyperText Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, widgets include XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! Widgets).

[0108] In conjunction with the RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, and browser module 147, widget creation module 150 can be used by a user to create widgets (e.g., turn user-specified portions of a web page into widgets).

[0109] In association with touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, search module 151 includes executable instructions for searching memory 102 for text, music, audio, images, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0110] In conjunction with touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that enable a user to download and play pre-recorded music and other audio files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing videos (e.g., on touchscreen 112 or on a display externally connected via external port 124). In some embodiments, device 100 may include the functionality of an MP3 player, such as an iPod (a registered trademark of Apple Inc.).

[0111] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, handwriting input module 157, and text input module 134, notes module 153 contains executable instructions for creating and managing notes, to-do lists, and the like, in accordance with user instructions.

[0112] In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, handwriting input module 157, text input module 134, GPS module 135, and browser module 147, map module 154 can be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data about businesses and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0113] In conjunction with touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, handwriting input module 157, text input module 134, email client module 140, and browser module 147, online video module 155 contains instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to the online videos, and otherwise manage the online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used to send links to particular online videos rather than email client module 140.

[0114] Each of the above-identified modules and applications corresponds to a set of executable instructions that perform one or more of the functions described above and methods described in the present application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various embodiments may combine or otherwise reconfigure various subsets of these modules. In some embodiments, memory 102 may store a subset of the above-identified modules and data structures. Additionally, memory 102 may store additional modules and data structures not described above.

[0115] In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively via a touchscreen and / or touchpad. By using the touchscreen and / or touchpad as the primary input control device for operation of device 100, the number of physical input control devices (push buttons, dials, etc.) on device 100 can be reduced.

[0116] FIG. 2 illustrates portable multifunction device 100 with touchscreen 112, according to some embodiments. The touchscreen can display one or more graphics within user interface (UI) 200. In this embodiment, as well as other embodiments described below, a user can select one or more of the graphics by performing a gesture on the graphics, for example, using one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of one or more graphics occurs when a user breaks contact with one or more graphics. In some embodiments, this gesture can include one or more taps, one or more swipes (left-to-right, right-to-left, upward, and / or downward), and / or rolling of a finger (right-to-left, left-to-right, upward, and / or downward) that has been in contact with device 100. In some embodiments, accidental contact with a graphic may not select that graphic. For example, a swipe gesture sweeping over an application icon may not select the corresponding application if the gesture corresponding to selection is a tap.

[0117] Device 100 may also include one or more physical buttons, such as a "home" or menu button 204. As previously mentioned, menu button 204 can be used to navigate to any application 136 within a set of applications that may run on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on touchscreen 112.

[0118] In one embodiment, device 100 includes touchscreen 112, menu button 204, pushbutton 206 for powering the device on / off and locking the device, volume control buttons 208, subscriber identity module (SIM) card slot 210, headset jack 212, and external docking / charging port 124. Pushbutton 206 can be used to turn power on / off on the device by pressing and holding the button down for a predetermined time interval, and to lock the device and / or unlock the device or begin the unlocking process by pressing and releasing the button before the predetermined time interval has elapsed. In an alternative embodiment, device 100 can also accept verbal input for activating or deactivating some features via microphone 113.

[0119] FIG. 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, a telephone device, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 may include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes input / output (I / O) interface 330 with display 340, which is typically a touchscreen display. I / O interface 330 may also include a keyboard and / or mouse (or other pointing device) 350, and a touchpad 355. Memory 370 may include high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access semiconductor memory devices, and may also include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile semiconductor storage devices. Memory 370 may optionally include one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1). Furthermore, memory 370 may store additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100.For example, memory 370 of device 300 may store drawing module 380, presentation module 382, ​​word processing module 384, website creation module 386, disc authoring module 388, and / or spreadsheet module 390, while memory 102 of portable multifunction device 100 (FIG. 1) may not store these modules.

[0120] Each of the above-identified elements of FIG. 3 can be stored in one or more of the memory devices mentioned above. Each of the above-identified modules corresponds to an instruction set that performs the described function. The above-identified modules or programs (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various embodiments may combine or otherwise reconfigure various subsets of these modules. In some embodiments, memory 370 may store a subset of the above-identified modules and data structures. Additionally, memory 370 may store additional modules and data structures not described above.

[0121] FIG. 4 shows an example user interface on a device (e.g., device 300 of FIG. 3 ) that includes a touch-sensitive surface 451 (e.g., tablet or touchpad 355 of FIG. 3 ) that is separate from a display 450 (e.g., touchscreen display 112). While many of the following examples are described with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. 4 . In some embodiments, this touch-sensitive surface (e.g., 451 in FIG. 4 ) has a major axis (e.g., 452 in FIG. 4 ) that corresponds to a major axis (e.g., 453 in FIG. 4 ) on the display (e.g., 450). According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4 ) at locations that correspond to respective locations on the display (e.g., 460 corresponds to 468, and 462 corresponds to 470 in FIG. 4 ). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462 and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4) of the multifunction device. It should be understood that similar methods can be used with respect to the other user interfaces described herein.

[0122] Attention is now directed to embodiments of handwriting input methods and user interfaces ("UIs") that may be implemented on a multifunction device (eg, device 100).

[0123] 5 is a block diagram illustrating an exemplary handwriting input module 157 that interacts with an I / O interface module 500 (e.g., I / O interface 330 in FIG. 3 or I / O subsystem 106 in FIG. 1) to provide handwriting input capabilities on the device, according to some implementations. As shown in FIG. 5, handwriting input module 157 includes an input processing module 502, a handwriting recognition module 504, and a result generation module 506. In some embodiments, input processing module 502 includes a segmentation module 508 and a normalization module 510. In some embodiments, result generation module 506 includes a radical clustering module 512 and one or more language models 514.

[0124] In some embodiments, input processing module 502 communicates with I / O interface module 500 (e.g., I / O interface 330 in FIG. 3 or I / O subsystem 106 in FIG. 1) to receive handwriting input from a user. The handwriting is input via any suitable means, such as touch-sensitive display system 112 in FIG. 1 and / or touchpad 355 in FIG. 3. The handwriting input includes data representing each stroke provided by the user within a predefined handwriting input area within the handwriting input UI. In some embodiments, the data representing each stroke of the handwriting input includes data such as the start and end locations, intensity profile, and path of movement of sustained contact (e.g., contact of the user's finger or stylus with the device's touch-sensitive surface) within the handwriting input area. In some embodiments, I / O interface module 500 passes an array 516 of handwriting strokes, with associated temporal and spatial information, to input processing module 502 in real time. At the same time, the I / O interface module also provides a real-time rendering 518 of the handwritten strokes within the handwriting input area of ​​the handwriting input user interface as visual feedback for this user input.

[0125] In some embodiments, as data representing each handwritten stroke is received by the input processing module 502, temporal and spatial information associated with multiple consecutive strokes is also recorded. For example, the data optionally includes a stack indicating the shape, size, spatial saturation, and relative spatial position of individual strokes, each with its stroke index, along the writing direction of the entire handwritten input. In some embodiments, the input processing module 502 returns instructions to the I / O interface module 500 to render the received strokes on the device's display 518 (e.g., display 340 in FIG. 3 or touch-sensitive display 112 in FIG. 1). In some embodiments, the rendering of the received strokes is animated to provide a visual effect that mimics the actual progression of writing on a writing surface (e.g., a sheet of paper) using a writing instrument (e.g., a pen). In some embodiments, the user is optionally allowed to specify the nib style, color, texture, etc. of the rendered strokes.

[0126] In some embodiments, input processing module 502 processes the strokes currently accumulated in the handwriting input area and assigns the strokes to one or more recognition units. In some embodiments, each recognition unit corresponds to a particular character recognized by handwriting recognition model 504. In some embodiments, each recognition unit corresponds to a particular output character or radical recognized by handwriting recognition model 504. A radical is a repeating component found within multiple compound logographic characters. A compound logographic character may contain two or more radicals arranged according to a common layout (e.g., left-to-right layout, top-to-bottom layout, etc.). In one example, a single Chinese character " JPEG2025124752000002.jpg1011" is composed of two radicals, namely the left radical " JPEG2025124752000003.jpg98" and the right radical " It is built using "JPEG2025124752000004.jpg1010".

[0127] In some embodiments, the input processing module 502 relies on a segmentation module to assign or separate currently stored handwritten strokes into one or more recognition units. For example, the handwritten character " When splitting strokes for "JPEG2025124752000005.jpg1011", splitting module 508 optionally splits strokes clustered on the left side of the handwritten input into a single radical (i.e., the left radical " JPEG2025124752000006.jpg98) and assigns strokes clustered on the right side of the handwriting input to a different recognition unit (i.e., the right radical " Alternatively, the segmentation module 508 may also segment all of these strokes into a single recognition unit (i.e., the character " It can also be assigned to a recognition unit (e.g., "JPEG2025124752000008.jpg1011").

[0128] In some embodiments, the segmentation module 508 segments the currently accumulated handwriting input (e.g., one or more handwritten strokes) into groups of recognition units in several different ways to create the segmentation lattice 520. For example, assume that a total of nine strokes have been accumulated in the handwriting input area so far. According to the first segmentation chain of the segmentation lattice 520, strokes 1, 2, and 3 are grouped into a first recognition unit 522, and strokes 4, 5, and 6 are grouped into a second recognition unit 526. According to the second segmentation chain of the segmentation lattice 520, all of strokes 1 through 9 are grouped into one recognition unit 526.

[0129] In some embodiments, each segmentation chain is given a segmentation score to assess the likelihood that the particular segmentation chain is a correct segmentation of the current handwriting input. In some embodiments, factors optionally used to calculate the segmentation score for each segmentation chain include the absolute and / or relative size of the strokes, the relative and / or absolute span of the strokes in various directions (e.g., x-direction, y-direction, z-direction), the average and / or variance of the saturation level of the strokes, the absolute and / or relative distance to adjacent strokes, the absolute and / or relative location of the strokes, the order or sequence in which the strokes are input, the duration of each stroke, the average and / or variance of the speed (or velocity) at which each stroke is input, the intensity profile of each stroke along its length, etc. In some embodiments, one or more functions or transforms are optionally applied to one or more of these factors to generate the segmentation scores for the various segmentation chains in segmentation bundle 520.

[0130] In some embodiments, after segmentation module 508 segments current handwriting input 516 received from a user, segmentation module 508 passes the segmentation lattice 520 to normalization module 510. In some embodiments, normalization module 510 generates an input image (e.g., input image 528) for each recognition unit specified in segmentation lattice 520 (e.g., recognition units 522, 524, and 526). In some embodiments, the normalization module performs any necessary or desired normalization (e.g., stretching, cropping, downsampling, or upsampling) on ​​the input image so that it can be provided as input to handwriting recognition model 504. In some embodiments, each input image 528 includes strokes assigned to a respective recognition unit and corresponds to a character or radical to be recognized by handwriting recognition module 504.

[0131] In some embodiments, the input image generated by the input processing module 502 does not include any temporal information associated with individual strokes; only spatial information (e.g., information represented by the location and density of pixels in the input image) is preserved in the input image. A handwriting recognition model trained purely on the spatial information of training writing samples is capable of handwriting recognition based solely on spatial information. As a result, the handwriting recognition model is stroke-order and stroke-direction independent and does not exhaustively enumerate all possible permutations of stroke order and stroke direction for all characters (i.e., all output classes) in the handwriting recognition model's vocabulary during training. Indeed, in some embodiments, the handwriting recognition module 502 does not distinguish between pixels belonging to one stroke and pixels belonging to another stroke in the input image.

[0132] As will be described in more detail later (e.g., in connection with Figures 25A-27), in some embodiments, some temporally derived stroke distribution information is reintroduced into a purely spatial handwriting recognition model to improve recognition accuracy without compromising the stroke order and stroke direction independence of that recognition model.

[0133] In some embodiments, the input image generated by the input processing module 502 for one recognition unit does not overlap with the input images of any other recognition units in the same segmentation chain. In some embodiments, the input images generated for different recognition units may have some overlap. In some embodiments, some overlap between input images is tolerated for recognizing handwritten input written in a cursive script and / or handwritten input that includes connected characters (e.g., one stroke connects two adjacent characters).

[0134] In some embodiments, some degree of normalization is performed before the segmentation. In some embodiments, the functions of the segmentation module 508 and the normalization module 510 may be performed by the same module or by two or more other modules.

[0135] In some embodiments, when the input image 528 of each recognition unit is provided as input to the handwriting recognition model 504, the handwriting recognition model 504 generates an output consisting of various likelihoods that the recognition unit is a respective output character within the repertoire or vocabulary of the handwriting recognition model 504 (i.e., a list of all characters and radicals recognizable by the handwriting recognition module 504). As described in more detail below, the handwriting recognition model 504 is trained to recognize a large number of characters in multiple scripts (e.g., at least three non-overlapping scripts encoded by the Unicode standard). Examples of non-overlapping scripts include Latin, Chinese, Arabic, Persian, Cyrillic, and artificial scripts such as Emoji. In some embodiments, the handwriting recognition model 504 generates one or more output characters for each input image (i.e., for each recognition unit) and assigns a respective recognition score for each output character based on a confidence level associated with the character recognition.

[0136] In some embodiments, the handwriting recognition model 504 generates a candidate lattice 530 according to the split lattice 520, and each arc in a split chain in the split lattice 520 (e.g., corresponding to a respective recognition unit 522, 524, 526) is expanded into one or more candidate arcs (e.g., arcs 532, 534, 536, 538, 540, each corresponding to a respective output character) in the candidate lattice 530. Each candidate chain in the candidate lattice 530 is scored according to the respective split scores of the split chains on which the candidate chain is based and the recognition scores associated with the output characters in the candidate chain.

[0137] In some embodiments, after the handwriting recognition model 504 generates output characters from the input images 528 of recognition units, the candidate lattice 530 is passed to a result generation module 506, which generates one or more recognition results for the currently accumulated handwriting input 516.

[0138] In some embodiments, result generation module 506 utilizes radical clustering module 512 to combine one or more radicals in the candidate lattice into compound characters. In some embodiments, result generation module 506 uses one or more language models 514 to determine whether the character chains in candidate lattice 530 are proper sequences in the particular language represented by the language model. In some embodiments, result generation module 506 generates corrected candidate lattice 542 by deleting particular arcs or combining two or more arcs in candidate lattice 530.

[0139] In some embodiments, result generation module 506 generates an integrated recognizability score for each character sequence (e.g., character sequences 544 and 546) still maintained in correction candidate lattice 542 based on the recognizability scores of the output characters in those character sequences as modified (e.g., increased or decreased) by radical clustering module 512 and language model 514. In some embodiments, result generation module 506 ranks the various character sequences maintained in correction candidate lattice 542 based on their integrated recognizability scores.

[0140] In some embodiments, the result generation module 506 sends the top-ranked character sequences to the I / O interface module 500 as ranked recognition results 548 for display to the user. JPEG2025124752000009.jpg814" and " JPEG2025124752000010.jpg1010") in the candidate display area of ​​the handwriting input interface. In some embodiments, the I / O interface module may display multiple recognition results (e.g., " JPEG2025124752000011.jpg814" and " JPEG2025124752000012.jpg1010") to the user and allows the user to select a recognition result and enter it as text input for the associated application. In some embodiments, the I / O interface module may select the top-ranked recognition result (e.g., " The recognition result is automatically entered based on other input or prompts from the user, such as "JPEG2025124752000013.jpg1010." The effective auto-entry of the top-ranked result can improve the efficiency of the input interface and provide a better user experience.

[0141] In some embodiments, result generation module 506 uses other factors to modify the combined recognition score of a candidate string. For example, in some embodiments, result generation module 506 optionally maintains a log of the most frequently used characters for a particular user or multiple users. Result generation module 506 optionally increases the combined recognition score of a particular candidate character or character sequence if the particular candidate character or character sequence is found in the list of most frequently used characters or character sequences.

[0142] In some embodiments, the handwriting input module 157 provides real-time updates regarding the recognition results displayed to the user. For example, in some embodiments, with each additional stroke by the user, the input processing module 502 optionally re-segments the currently accumulated handwriting input and corrects the segmentation lattice and input image provided to the handwriting recognition model 504. Similarly, the handwriting recognition model 504 optionally corrects the candidate lattice provided to the result generation module 506. As a result, the result generation module 506 optionally updates the recognition results presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user immediately or within a short period of time (e.g., within tens of milliseconds to a few seconds). Real-time handwriting recognition differs from offline recognition (e.g., as in offline optical character recognition (OCR) applications) in that recognition begins immediately and is performed substantially simultaneously with the receipt of handwritten input, rather than being performed all at once after the current user session from a recorded image that is saved for later retrieval. Furthermore, because offline character recognition is performed without any temporal information about individual strokes and stroke sequences, segmentation is performed without the benefit of such information. Further disambiguation between similar-looking candidate characters also does not benefit from such temporal information.

[0143] In some embodiments, the handwriting recognition model 504 is implemented as a convolutional neural network (CNN). Figure 6 shows an example convolutional neural network 602 trained on a multi-script training corpus 604 containing writing samples for characters in multiple non-overlapping scripts.

[0144] As shown in FIG. 6 , the convolutional neural network 602 includes an input plane 606 and an output plane 608. Between the input plane 606 and the output plane 608 are multiple convolutional layers 610 (e.g., including a first convolutional layer 610 a, zero or more intermediate convolutional layers (not shown), and a final convolutional layer 610 n). Each convolutional layer 610 is followed by a respective subsampling layer 612 (e.g., a first subsampling layer 612 a, zero or more intermediate subsampling layers (not shown), and a final subsampling layer 612 n). After these convolutional and subsampling layers, just before the output plane 608, is a hidden layer 614. The hidden layer 614 is the final layer before the output plane 608. In some embodiments, a kernel layer 616 (e.g., including a first kernel layer 616 a, zero or more intermediate kernel layers (not shown), and a final kernel layer 612 n) is inserted before each convolutional layer 610 to improve computational efficiency.

[0145] As shown in Figure 6, input surface 606 receives an input image 614 of a handwritten recognition unit (e.g., a handwritten character or radical), and output surface 608 outputs a set of probabilities indicating the likelihood that the recognition unit belongs to a corresponding output class (e.g., a particular character in the set of output characters that the neural network is configured to recognize). The entire set of output classes of the neural network (or the set of output characters of the neural network) is also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural networks described herein can be trained to have a repertoire of tens of thousands of characters.

[0146] As the input image 614 is processed through various layers of the neural network, various spatial features embedded in the input image 614 are extracted by the convolutional layer 610. Each convolutional layer 610, also referred to as a set of feature maps, acts as a filter to find specific features in the input image 614 to distinguish images corresponding to different characters. The subsampling layer 612 ensures that features are captured at gradually increasing scales from the input image 614. In some embodiments, the subsampling layer 612 is implemented using a max-pooling technique. This max-pooling layer creates position invariance over larger local regions and downsamples the output image of the previous convolutional layer by a factor of Kx and Ky along each direction (Kx and Ky are the sizes of the max-pooling rectangle). Max-pooling results in faster convergence and improved generalization by selecting top-invariant features. In some embodiments, subsampling is achieved using other methods.

[0147] In some embodiments, after the convolutional layer 610n and the final set of subsampling 612n, and before the output plane 608, there is a fully connected layer, i.e., hidden layer 614. The fully connected hidden layer 614 is a multi-layer perceptron that fully connects the nodes in the final subsampling layer 612n to the nodes in the output plane 608. The hidden layer 614 takes the output image received from the previous layer and, through logistic regression, arrives at one of the output characters in the output layer 608.

[0148] During training of the convolutional neural network 602, the features and their associated weights in the convolutional layers 610, as well as the weights associated with the parameters in the hidden layers 614, are adjusted to minimize classification errors with respect to writing samples with known output classes in the training corpus 604. Once the convolutional neural network 602 is trained and optimal sets of parameters and associated weights are established for the various layers in the network, the convolutional neural network 602 can be used to recognize new writing samples 618 that are not part of the training corpus 604, such as input images generated based on real-time handwriting input received from a user.

[0149] As described herein, the convolutional neural network of the handwriting input interface is trained using a multi-script training corpus to enable multi-script or mixed-script handwriting recognition. In some embodiments, the convolutional neural network is trained to recognize a wide repertoire of over 30,000 to 60,000 characters (e.g., all characters encoded by the Unicode standard). Most state-of-the-art handwriting recognition systems are based on stroke-order-dependent hidden Markov models (HMMs). Furthermore, most existing handwriting recognition models are language-specific, encompassing small repertoires of a few dozen characters (e.g., characters from the English alphabet, the Greek alphabet, all ten numerals, etc.) up to several thousand characters (e.g., the most commonly used Chinese character set). Therefore, the general-purpose recognizer described herein can handle several orders of magnitude more characters than most existing systems.

[0150] Some conventional handwriting systems may include several individually trained handwriting recognition models, each tailored for a particular language or small set of characters. A writing sample is propagated through various recognition models until it can be classified. For example, a handwriting sample may be provided to a series of concatenated language- or script-specific character recognition models; if the handwriting sample cannot be conclusively classified by the first recognition model, it is provided to the next recognition model, which attempts to classify the handwriting sample within its own repertoire. This approach to classification is time consuming, and memory requirements grow rapidly with each additional recognition model that needs to be employed.

[0151] Other state-of-the-art models require the user to specify a preferred language, and then use the selected handwriting recognition model to classify the current input. Such implementations are not only cumbersome to use and consume significant memory, but also cannot be used to recognize mixed-language input. It is impractical for users to be required to switch language settings in the middle of mixed-language or mixed-script input.

[0152] The multi-script or generic recognizer described herein addresses at least some of the above-noted problems with conventional recognition systems. Figure 7 is a flow diagram of an exemplary process 700 for using a large multi-script training corpus to train a handwriting recognition module (e.g., a convolutional neural network) that can then be used to provide real-time multi-language and multi-script handwriting recognition of a user's handwritten input.

[0153] In some embodiments, training of the handwriting recognition model is performed on a server device, and the trained handwriting recognition model is then provided to the user device. The handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without further assistance from the server. In some embodiments, both training and recognition are provided on the same device. For example, the server device can receive user handwriting input from the user device, perform handwriting recognition, and transmit the recognition results to the user device in real time.

[0154] In exemplary process 700, a device having one or more processors and memory trains a multiscript handwriting recognition model based on spatially derived features (e.g., stroke-order-independent features) of a multiscript training corpus (702). In some embodiments, the spatially derived features of the multiscript training corpus are stroke-order-independent and stroke-direction-independent (704). In some embodiments, training of the multiscript handwriting recognition model is independent of temporal information associated with each stroke in a handwriting sample (706). Specifically, images of the handwriting sample are normalized to a predetermined size, and the images do not contain any information about the order in which the individual strokes were input to form the image. Furthermore, the images also do not contain any information about the direction in which the individual strokes were input to form the image. Indeed, during training, features are extracted from the handwriting images regardless of how the individual strokes form the image in time. Therefore, temporal information associated with the individual strokes is not required during recognition. As a result, the recognition robustly provides consistent recognition results despite delayed, out-of-order strokes and arbitrary stroke directions within the handwritten input.

[0155] In some embodiments, the multi-script training corpus includes handwriting samples corresponding to characters of at least three non-overlapping scripts. As shown in Figure 6, the multi-script training corpus includes handwriting samples collected from many users. Each handwriting sample corresponds to one character of each script represented in the handwriting recognition model. To adequately train the handwriting recognition model, the training corpus includes a large number of writing samples for each character of the scripts represented in the handwriting recognition model.

[0156] In some embodiments, the at least three non-overlapping scripts include Chinese, Emoji, and Latin scripts (708). In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing 30,000 characters across the at least three non-overlapping scripts (710).

[0157] In some embodiments, the multiscript training corpus includes writing samples for each character in all Chinese characters encoded in the Unicode Standard (e.g., all or a substantial portion of all CJK (Chinese-Japanese-Korean) Unified Ideographs). The Unicode Standard defines a total of approximately 74,000 CJK Unified Ideographs. The CJK Unified Ideographs basic block (4E00-9FFF) includes 20,941 basic Chinese characters used in Chinese, as well as Japanese, Korean, and Vietnamese. In some embodiments, the multiscript training corpus includes writing samples for all characters in the CJK Unified Ideographs basic block. In some embodiments, the multiscript training corpus further includes writing samples for CJK radicals that can be used to structurally compose one or more compound Chinese characters. In some embodiments, the multiscript training corpus further includes writing samples for less frequently used Chinese characters, such as Chinese characters encoded in one or more CJK Unified Ideograph extensions.

[0158] In some embodiments, the multiscript training corpus further includes writing samples for each character in the Latin script, including all characters in the Latin script encoded by the Unicode standard. Characters in the basic Latin script include the Latin uppercase and lowercase letters, as well as basic symbols and numbers commonly used on standard Latin keyboards. In some embodiments, the multiscript training corpus further includes characters in the extended Latin script (e.g., various stress forms of the basic Latin characters).

[0159] In some embodiments, the multi-script training corpus includes writing samples corresponding to each character of an artificial script that is not associated with any natural human language. For example, in some embodiments, a set of emojis is optionally defined in an emoji script, and writing samples corresponding to each of those emojis are included in the multi-script training corpus. For example, a hand-drawn heart symbol is represented by the emoji " in the training corpus. Similarly, the handwritten smiley face (e.g., two dots above an upward-pointing arc) is a handwritten example of the emoji " JPEG2025124752000015.jpg1110. Other emoji include categories of icons representing various emotions (e.g., joy, sadness, anger, embarrassment, surprise, laughter, crying, annoyance, etc.), various objects and characters (e.g., cats, dogs, rabbits, hearts, fruits, eyes, lips, gifts, flowers, candles, moons, stars, etc.), and various actions (e.g., shaking hands, kissing, running, dancing, jumping, sleeping, eating, meeting, loving, liking, voting, etc.). In some embodiments, the strokes in the handwriting sample corresponding to an emoji are simplified and / or stylized versions of the actual lines that form the corresponding emoji. In some embodiments, each device or application may use a different design for the same emoji. For example, a smiling emoji presented to a female user may be different from a smiling emoji presented to a male user, even if the handwritten input received from the two users is substantially the same.

[0160] In some embodiments, the multiscript training corpus also includes writing samples for characters in other scripts, such as Greek script (e.g., including Greek letters and symbols), Cyrillic script, Hebrew script, and one or more other scripts encoded according to the Unicode standard. In some embodiments, the at least three non-overlapping scripts included in the multiscript training corpus include characters in Chinese, Emoji, and Latin scripts. Characters in Chinese, Emoji, and Latin scripts are naturally non-overlapping scripts. Many other scripts may overlap with each other, at least with respect to some characters. For example, some characters in Latin script (e.g., A, Z) can be found in many other scripts (e.g., Greek and Cyrillic). In some embodiments, the multiscript training corpus includes Chinese, Arabic, and Latin scripts. In some embodiments, the multiscript training corpus includes other combinations of overlapping and / or non-overlapping scripts. In some embodiments, the multiscript training corpus includes writing samples for all characters encoded by the Unicode standard.

[0161] As shown in FIG. 7 , in some embodiments, to train a multiscript handwriting recognition model, a device provides handwriting samples from a multiscript training corpus to a single convolutional neural network having a single input plane and a single output plane (712). The device uses the convolutional neural network to determine spatially derived features (e.g., stroke-order-independent features) of the handwriting samples and respective weights for those spatially derived features for distinguishing between characters of at least three non-overlapping scripts represented in the multiscript training corpus (714). This multiscript handwriting recognition model differs from conventional multiscript handwriting recognition models in that a single handwriting recognition model having a single input plane and a single output plane is trained using all samples in the multiscript training corpus. The single convolutional neural network is trained to identify all characters represented in the multiscript training corpus without relying on separate subnetworks (e.g., subnetworks each trained on a particular script or character used in a particular language), each of which handles a small subset of the training corpus. Furthermore, this single convolutional neural network is trained to identify a large number of characters across multiple non-overlapping scripts, rather than characters from a small number of overlapping scripts, such as Latin and Greek scripts (e.g., with overlapping letters A, B, E, Z, etc.).

[0162] In some embodiments, the device provides real-time handwriting recognition of the user's handwritten input using a multi-script handwriting recognition model that has been trained on spatially derived features of the multi-script training corpus (716). In some embodiments, providing real-time handwriting recognition of the user's handwritten input includes continually correcting the recognition output for the user's handwritten input as the user continues to provide additions and corrections to the handwritten input. In some embodiments, providing real-time handwriting recognition of the user's handwritten input further includes providing the multi-script handwriting recognition model to a user device (718), where the user device receives the handwritten input from the user and locally performs handwriting recognition on the handwritten input based on the multi-script handwriting recognition model.

[0163] In some embodiments, a device provides a multi-script handwriting recognition model to multiple devices, each with no overlap in its input language, and the multi-script handwriting recognition model is used on each of the multiple devices for handwriting recognition in different languages ​​associated with each user device. For example, if a multi-script handwriting recognition model is trained to recognize characters in many different scripts and languages, the same handwriting recognition model can be used worldwide to provide handwriting input for any of the input languages. A first device for a user who only wants to input in English and Hebrew can provide handwriting input functionality using the same handwriting recognition model as a second device for a user who only wants to input in Chinese and Emoji. Instead of requiring a user of a first device to separately install an English handwriting input keyboard (e.g., implemented with an English-specific handwriting recognition model) and a separate Hebrew handwriting input keyboard (e.g., implemented with a Hebrew-specific handwriting recognition model), the same general-purpose multi-script handwriting recognition model, once installed on the first device, can be used to provide handwriting input functionality for both English and Hebrew, as well as mixed input in both languages. Furthermore, instead of requiring the second user to install a Chinese handwriting input keyboard (e.g., implemented with a Chinese-specific handwriting recognition model) and a separate emoji handwriting input keyboard (e.g., implemented with an emoji handwriting recognition model), the same general-purpose multi-script handwriting recognition model, once installed on the second device, can be used to provide handwriting input capabilities for both Chinese and emoji, as well as mixed input in both scripts. Using the same multi-script handwriting model to handle a wide repertoire across multiple scripts (e.g., a substantial portion or all of the characters encoded in nearly 100 different scripts) increases the usefulness of the recognizer without imposing a substantial burden on device providers and users.

[0164] Training a multiscript handwriting recognition model using a large multiscript training corpus differs from traditional HMM-based handwriting recognition systems in that it does not rely on temporal information associated with individual strokes of characters. Furthermore, the resource and memory requirements for this multiscript recognition system do not increase proportionally with the increase in symbols and languages ​​covered by the multiscript recognition system. For example, in traditional handwriting systems, increasing the number of languages ​​means adding another independently trained model, and memory requirements at least double to accommodate the increased capabilities of the handwriting recognition system. In contrast, when a multiscript model is trained with a multiscript training corpus, increasing language coverage requires retraining the handwriting recognition model using additional handwriting samples, increasing the size of the output surface, but by a very modest amount. Suppose the multiscript training corpus includes handwriting samples corresponding to n different languages, and the multiscript handwriting recognition model occupies a memory of size m and increases language coverage to N (N>n) languages. The device retrains the multiscript handwriting recognition model based on spatially derived features of a second multiscript training corpus, the second multiscript training corpus including second handwriting samples corresponding to N different languages. When N / n varies from 1 to 100, the change in M / m remains substantially constant within a range of 1 to 2. After the multiscript handwriting recognition model is retrained, the device can provide real-time handwriting recognition of a user's handwritten input using the retrained multiscript handwriting recognition model.

[0165] 8A and 8B illustrate an exemplary user interface for providing real-time multi-script handwriting recognition and input on a portable user device (e.g., device 100). In FIGS. 8A and 8B, a handwriting input interface 802 is displayed on a touch-sensitive display screen (e.g., touchscreen 112) of the user device. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text input area 808. In some embodiments, the handwriting input interface 802 further includes a plurality of control elements, each of which can be invoked to cause the handwriting input interface to perform a predetermined function. As shown in FIG. 8A, included within the handwriting input interface are a delete button, a space button, a carriage return or enter button, and a keyboard toggle button. Other control elements are possible and can optionally be provided within the handwriting input interface to suit different applications utilizing the handwriting input interface 802. The layout of the various components of the handwriting input interface 802 is merely exemplary and may vary for different devices and different applications.

[0166] In some embodiments, the handwriting input area 804 is a touch-sensitive area for receiving handwritten input from a user. A sustained contact on the touchscreen within the handwriting input area 804 and its associated path of movement are registered as a handwritten stroke. In some embodiments, the handwritten strokes registered by the device are visually rendered within the handwriting input area 804 at the same location traced by the sustained contact. As shown in FIG. 8A , the user can write several handwritten Chinese characters (e.g., " )" within the handwriting input area 804. JPEG2025124752000016.jpg1019), some handwritten English characters (e.g., "Happy"), and handwritten emoticons (e.g., a smiley face). These handwritten characters are distributed across multiple lines (e.g., two lines) in the handwriting input area 804.

[0167] In some embodiments, the candidate display area 806 displays one or more recognition results (e.g., 810 and 812) for the handwritten input currently stored in the handwriting input area 804. Typically, the top-ranked recognition result (e.g., 810) is displayed in the first position in the candidate display area. As shown in FIG. 8A , the handwriting recognition model described herein is capable of recognizing characters in multiple non-overlapping scripts, including Chinese characters, Latin scripts, and emojis, so the recognition results (e.g., 810) provided by this recognition model accurately include the Chinese characters, English characters, and emojis represented by the handwritten input. The user is not required to interrupt mid-writing of the input to select or switch the recognition language.

[0168] In some embodiments, the text entry area 808 is an area that displays text input provided to each application employing this handwriting input interface. As shown in FIG. 8A, the text entry area 808 is used by a notes application to display the text currently displayed in the text entry area 808 (e.g., "America JPEG2025124752000017.jpg1029") is the text input already provided to the notes application. In some embodiments, cursor 813 indicates the current text entry position within text entry area 808.

[0169] In some embodiments, the user can select a particular recognition result displayed in the candidate display area 806, for example, by an explicit selection input (e.g., a tap gesture on one of the displayed recognition results) or an implicit confirmation input (e.g., a tap gesture on the “Enter” button or a double-tap gesture in the handwriting input area). As shown in FIG. 8B , the user explicitly selects the top-ranked recognition result 810 using a tap gesture (as indicated by contact 814 on recognition result 810 in FIG. 8A ). In response to this selection input, the text of recognition result 810 is inserted into the text input area 808 at an insertion point indicated by cursor 813. As shown in FIG. 8B , once the text of the selected recognition result 810 has been entered into the text input area 808, both the handwriting input area 804 and the candidate display area 806 are cleared. At this point, the handwriting input area 804 is ready to accept new handwriting input, and the candidate display area 806 is now available to display recognition results for the new handwriting input. In some embodiments, implicit confirmation causes the top-ranked recognition result to be entered into the text input area 808 without requiring the user to pause and select the top-ranked recognition result. Well-designed implicit confirmation increases text entry speed and reduces the cognitive burden imposed on the user during text composition.

[0170] In some embodiments (not shown in FIGS. 8A and 8B ), the top-ranked recognition result of the current handwriting input is optionally provisionally displayed in the text input area 808. The provisional text input shown in the text input area 808 is visually distinguished from other text inputs in the text input area, for example, by a provisional input box surrounding the provisional text input. The text shown in the provisional input box has not yet been finalized or provided to an associated application (e.g., a notes application), and is automatically updated when the top-ranked recognition result is changed by the handwriting input module, for example, in response to user corrections of the current handwriting input.

[0171] 9A and 9B are a flow diagram of an example process 900 for providing multi-script handwriting recognition on a user device. In some embodiments, as shown in diagram 900, the user device receives (902) a multi-script handwriting recognition model that has been trained on spatially derived features (e.g., stroke order and stroke direction independent features) of a multi-script training corpus that includes handwriting samples corresponding to characters of at least three non-overlapping scripts. In some embodiments, the multi-script handwriting recognition model is a single convolutional neural network (906) with a single input plane and a single output plane that includes spatially derived features and respective weights for those spatially derived features for distinguishing characters of at least three non-overlapping scripts represented in the multi-script training corpus. In some embodiments, the multi-script handwriting recognition model is configured to recognize characters based on respective input images of one or more recognition units identified in the handwritten input (908), and the respective spatially derived features used for recognition are independent of the respective stroke order, stroke direction, and stroke continuity within the handwritten input.

[0172] In some embodiments, the user device receives (908) handwriting input from a user, the handwriting input including one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device. For example, the handwriting input includes data about the location and movement of contact between a finger or a stylus and the touch-sensitive surface coupled to the user device. In response to receiving the handwriting input, the user device provides (910)(912) one or more handwriting recognition results to the user in real time based on a multi-script handwriting recognition model that has been trained on spatially derived features of a multi-script training corpus.

[0173] In some embodiments, when providing real-time handwriting recognition results to the user, the user device segments (914) the user's handwriting input into one or more recognition units, each recognition unit including one or more of the handwriting strokes provided by the user. In some embodiments, the user device segments the user's handwriting input according to the shape, location, and size of the individual strokes created by the user's finger or stylus contacting the touch-sensitive surface of the user device. In some embodiments, this segmentation of the handwriting input further takes into account the relative order and relative position of the individual strokes created by the user's finger or stylus contacting the touch-sensitive surface of the user device. In some embodiments, the user's handwriting input is in a cursive script, and each successive stroke in the handwriting input may correspond to multiple strokes in a recognized character in printed form. In some embodiments, the user's handwriting input may include successive strokes across multiple recognized characters in printed form. In some embodiments, segmenting the handwriting input generates one or more input images, each corresponding to a respective recognition unit. In some embodiments, some of the input images optionally include some overlapping pixels. In some embodiments, the input images do not contain any overlapping pixels. In some embodiments, the user device generates a segmentation lattice, where each segmentation chain in the segmentation lattice represents a different way to segment the current handwritten input. In some embodiments, each arc in the segmentation chain corresponds to a different group of strokes in the current handwritten input.

[0174] As shown in diagram 900, the user device provides corresponding images of one or more recognition units as input to the multiscript recognition model (914). For at least one of the one or more recognition units, the user device obtains from the multiscript handwriting recognition model at least a first output character from a first script and at least a second output character from a second script different from the first script (916). For example, the same input image can cause the multiscript recognition model to output two or more similar-looking output characters from different scripts as recognition results for the same input image. For example, the handwriting input for the letter "a" in Latin script and the letter "α" in Greek script are often similar. Furthermore, the handwriting input for the letter "J" in Latin script and the Chinese character " Handwritten inputs for "JPEG2025124752000018.jpg89" are often similar. Handwritten input for "JPEG2025124752000019.jpg1110" can be done using the CJK radical " The handwriting input may resemble the handwriting input for "JPEG2025124752000020.jpg910." In some embodiments, the multi-script handwriting recognition model often generates multiple candidate recognition results that likely correspond to the user's handwriting input because the visual appearance of the handwriting input makes it difficult for even a human reader to decipher. In some embodiments, the first script is a CJK base character block and the second script is a Latin script as encoded by the Unicode standard. In some embodiments, the first script is a CJK base character block and the second script is a set of emoji characters. In some embodiments, the first script is a Latin script and the second script is emoji characters.

[0175] In some embodiments, the user device displays both the first output character and the second output character in a candidate display area of ​​a handwriting input interface of the user device (918). In some embodiments, the user device selectively displays one of the first output character and the second output character based on which one of the first script and the second script is the corresponding script used in a soft keyboard currently installed on the user device (920). For example, if the handwriting recognition model selects the Chinese character " _ ... Assuming that the user has specified "JPEG2025124752000021.jpg910" and the Greek character "λ", the user device determines whether the user has installed on the user device a Chinese soft keyboard (e.g., a keyboard using the Pinyin input method) or a Greek input keyboard. If the user device determines that only a Chinese soft keyboard is installed, the user device optionally displays the Chinese character " JPEG2025124752000022.jpg910" and does not display the Greek letter "λ".

[0176] In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, the user device continuously corrects one or more recognition results for a user's handwriting input in response to the user's continuous additions or corrections to the handwriting input before the user makes an explicit or implicit selection of a recognition result displayed to the user (922). In some embodiments, in response to each correction of one or more recognition results, the user displays (924) each corrected recognition result or results to the user in a candidate display area of ​​a handwriting input user interface.

[0177] In some embodiments, the multi-script handwriting recognition model is trained to recognize all characters in at least three non-overlapping scripts, including Chinese, Emoji, and Latin scripts, as encoded according to the Unicode standard (926). In some embodiments, the at least three non-overlapping scripts include Chinese, Arabic, and Latin scripts. In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing at least 30 characters across the at least three non-overlapping scripts (928).

[0178] In some embodiments, the user device allows a user to enter multi-script handwriting input, such as a phrase that includes characters from more than one script. For example, a user can write continuously and receive handwriting recognition results that include characters from more than one script, without interrupting mid-writing to manually switch the recognition language. For example, a user can enter the multi-script sentence "Hello means" in the handwriting input area of ​​the user device. JPEG2025124752000023.jpg1120in Chinese.', in which case the Chinese character ' JPEG2025124752000024.jpg1120" or switching the input language from Chinese back to English when writing the English words "in Chinese".

[0179] As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition of a user's input. In some embodiments, the real-time handwriting recognition is used to provide real-time multi-script handwriting input functionality on a user's device. Figures 10A-10C are flow diagrams of an example process 1000 for providing real-time handwriting recognition and input on a user device. Specifically, the real-time handwriting recognition is stroke-order independent at the character, phrase, and sentence levels.

[0180] In some embodiments, stroke-order-independent handwriting recognition at the character level requires that a handwriting recognition model provide the same recognition results for a particular handwritten character regardless of the arrangement of the individual strokes of that character provided by the user. For example, the individual strokes of Chinese characters are typically written in a particular order. Native Chinese speakers are often trained in school to write each character in a particular order, but many users later adopt personal styles and stroke arrangements that deviate from that traditional stroke order. Furthermore, cursive writing styles are highly individualized, and multiple strokes in the printed form of a Chinese character are often fused into a single stylized stroke that may be distorted, redirected, or even continue onto the next character. A stroke-order-independent recognition model is trained based on images of writing samples that do not include temporal information associated with the individual strokes. Therefore, the recognition is independent of stroke order information. For example, the Chinese character " For "JPEG2025124752000025.jpg1010", the same recognition result "" is obtained regardless of whether the user writes the horizontal stroke first or the vertical stroke first. JPEG2025124752000026.jpg1010" is given by the handwriting recognition model.

[0181] As shown in FIG. 10A, in process 1000, a user device receives (1002) a plurality of handwritten strokes from a user, where the plurality of handwritten strokes corresponds to a single handwritten character. For example, the character " Handwritten input for "JPEG2025124752000027.jpg1010" typically includes substantially horizontal handwritten strokes intersecting with substantially vertical handwritten strokes.

[0182] In some embodiments, the user device generates an input image based on the plurality of handwritten strokes (1004). In some embodiments, the user device provides the input image to a handwriting recognition model (1006) to perform real-time handwriting recognition of the handwritten character, the handwriting recognition model providing stroke-order-independent handwriting recognition. The user device then generates an identical first output character (e.g., the character " in printed form) in real time after receiving the plurality of handwritten strokes (e.g., horizontal strokes and vertical strokes) from the user, regardless of their respective order. Display the image (1008) containing the image file "JPEG2025124752000028.jpg1010".

[0183] In some conventional handwriting recognition systems, minor stroke order variations in a small number of characters are tolerated by specifically including such variations in the training of the handwriting recognition system. Such conventional handwriting recognition systems are not scalable to accommodate arbitrary stroke order variations in a large number of complex characters, such as Chinese characters, because even characters of moderate complexity already result in a large number of stroke order variations. Furthermore, simply including more permutations of acceptable stroke orders for a particular character still leaves conventional recognition systems unable to handle handwritten input in which multiple strokes are combined into a single stroke (e.g., as in hypercursive writing) or in which a stroke is decomposed into multiple substrokes (e.g., as in characters captured with a very coarse sampling of the input strokes). Therefore, a multi-script handwriting system trained on spatially derived features, as described herein, offers advantages over conventional recognition systems.

[0184] In some embodiments, stroke-order-independent handwriting recognition is performed independently of temporal information associated with individual strokes within each handwritten character. In some embodiments, stroke-order-independent handwriting recognition is performed in conjunction with stroke distribution information that takes into account the spatial distribution of individual strokes before they are fused into a flat input image. Further details on how this temporally derived stroke distribution information is used to enhance the stroke-order-independent handwriting recognition described above are provided later in this specification (e.g., in conjunction with Figures 25A-27). The techniques described in conjunction with Figures 25A-27 do not compromise the stroke-order independence of the handwriting recognition system.

[0185] In some embodiments, the handwriting recognition model provides stroke-direction-independent handwriting recognition (1010). In some embodiments, stroke-direction-independent recognition requires that the user device, in response to receiving multiple handwritten inputs, displays the same first output character regardless of the respective stroke direction of each of the multiple handwritten strokes provided by the user. For example, if a user writes the Chinese character " " in the handwriting input area of ​​the user device, For example, if a user is writing "JPEG2025124752000029.jpg1010," the handwriting recognition model will output the same recognition result whether the user draws horizontal strokes from left to right or from right to left. Similarly, the handwriting recognition model will output the same recognition result whether the user draws vertical strokes in a downward or upward direction. In another example, many Chinese characters are structurally created with two or more radicals. Some Chinese characters include a left radical and a right radical, and people conventionally write the left radical first and the right radical second. In some embodiments, the handwriting recognition model will provide the same recognition result whether the user writes the left radical first or the right radical first, as long as the resulting handwritten input shows the left radical to the left of the right radical when the user completes the handwritten character. Similarly, some Chinese characters include an upper radical and a lower radical, and people conventionally write the upper radical first and the lower radical last. In some embodiments, the handwriting recognition model provides the same recognition results regardless of whether the user writes the upper radical first or the lower radical first, as long as the resulting handwritten input shows the upper radical above the lower radical. In other words, the handwriting recognition model determines the identity of handwritten characters without relying on the user-provided direction of the individual strokes of the handwritten character.

[0186] In some embodiments, the handwriting recognition model provides handwriting recognition based on the image of the recognition unit regardless of the number of substrokes of the recognition unit provided by the user. In other words, in some embodiments, the handwriting recognition model provides stroke-number-independent handwriting recognition (1014). In some embodiments, the user device, in response to receiving multiple handwritten strokes, displays the same first output character regardless of how many handwritten strokes are used to form successive strokes in the input image. For example, if the user is writing the Chinese character "+" in the handwriting input area, the handwriting recognition model may determine that the user has written the character " It will produce the same recognition result whether you provide four strokes to form the shape of "JPEG2025124752000030.jpg1010" (e.g., two short horizontal strokes and two short vertical strokes to form a cross-shaped character), or two strokes (e.g., an L-shaped stroke and a 7-shaped stroke, or a horizontal stroke and a vertical stroke), or any other number of strokes (e.g., hundreds of extremely short strokes or dots).

[0187] In some embodiments, not only is the handwriting recognition model capable of recognizing the same character regardless of the order, direction, and number of strokes in which each single character is written, but the handwriting recognition model is also capable of recognizing multiple characters regardless of the temporal order of the strokes of the characters provided by the user.

[0188] In some embodiments, the user device not only receives a first plurality of handwritten strokes from the user, but also receives a second plurality of handwritten strokes (1016), where the second plurality of handwritten strokes correspond to a second handwritten character. In some embodiments, the user device generates a second input image based on the second plurality of handwritten strokes (1018). In some embodiments, the user device provides the second input image to a handwriting recognition model (1020) to perform real-time recognition of the second handwritten character. In some embodiments, the user device displays a second output character corresponding to the second plurality of handwritten strokes in real time as the second plurality of handwritten strokes are received (1022). In some embodiments, the second output character and the first output character are simultaneously displayed in a spatial arrangement independent of the respective order of the first plurality of handwritten strokes and the second plurality of handwritten strokes provided by the user. For example, if a user writes two Chinese characters (e.g., " )" in a handwriting input area of ​​the user device, JPEG2025124752000031.jpg1010" and " JPEG2025124752000032.jpg1011), the user device determines that the handwriting currently stored in the handwriting input area is the character " To the left of the stroke for "JPEG2025124752000033.jpg1011", the character " As long as the user indicates the strokes related to "JPEG2025124752000034.jpg1010", JPEG2025124752000035.jpg1010" was first written, or the character " Regardless of whether the stroke of "JPEG2025124752000036.jpg1011" was written first, the recognition result " JPEG2025124752000037.jpg1019 is displayed. Before some of the strokes (e.g., vertical strokes) in "JPEG2025124752000038.jpg1010", the characters " Even if you write some of the strokes (e.g., strokes tilted to the left) for "JPEG2025124752000039.jpg1011", the resulting image of the handwriting input in the handwriting input area will be the character " To the left of all strokes related to "JPEG2025124752000040.jpg1011", the character " As long as the user device shows all the strokes related to "JPEG2025124752000041.jpg1010", the user device can obtain the recognition result " "JPEG2025124752000042.jpg1019".

[0189] 10B, in some embodiments, the spatial arrangement of the first output character and the second output character corresponds to the spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along a default (e.g., left-to-right) writing direction of a handwriting input interface of the user device (1024). In some embodiments, the second plurality of handwritten strokes is received 1026 after the first plurality of handwritten strokes in time, and the second output character precedes the first output character in the spatial arrangement along the default (e.g., left-to-right) writing direction of the handwriting input interface of the user device.

[0190] In some embodiments, the handwriting recognition model provides stroke-order independent recognition at the sentence-by-sentence level. For example, within a handwriting input area, the handwritten character " JPEG2025124752000043.jpg1010" is present in the first handwritten sentence, and the handwritten characters " Even if "JPEG2025124752000044.jpg1011" is present in a second handwritten sentence and the two handwritten characters are separated by one or more other handwritten characters and / or words, the handwriting recognition model will still be able to separate the two characters into the spatial sequence " JPEG2025124752000045.jpg1031". This recognition result and the spatial arrangement of the two recognized characters are such that when the user completes the handwriting input, the recognition units for the two characters are arranged in the array " JPEG2025124752000046.jpg1031", the spatial arrangement of the strokes of those two characters will remain the same regardless of the temporal order of the strokes of those two characters provided by the user. JPEG2025124752000047.jpg1010") is a first handwritten sentence (e.g., " "JPEG2025124752000048.jpg1010is a number.") and a second handwritten character (e.g., " JPEG2025124752000049.jpg1011") is a second handwritten sentence (e.g., " JPEG2025124752000050.jpg1011 is another number."), and the first and second handwritten sentences are simultaneously displayed in the handwriting input area of ​​the user device. In some embodiments, the recognition result (e.g., " JPEG2025124752000051.jpg1010is a number. Once the user confirms that "JPEG2025124752000052.jpg1011 is another number." is the correct recognition result, the two sentences are entered into the text entry area of ​​the user device, and the handwriting entry area is cleared for the user to enter another handwriting entry.

[0191] In some embodiments, the handwriting recognition model is stroke-order independent not only at the character level but also at the word and sentence levels, allowing a user to make corrections to a previously incomplete character after writing a subsequent character. For example, if a user forgets to write a particular stroke for a character before moving on to writing one or more subsequent characters in the handwriting input area, the user can still later fill in the missing stroke in the correct place within the particular character and receive the correct recognition result.

[0192] In conventional stroke-order-dependent systems (e.g., HMM-based recognition systems), once a character is written, the character is finalized and the user can no longer make any changes to the character. If the user wants to make any changes, the user must delete the character and all subsequent characters and start over. Some conventional recognition systems require the user to complete a handwritten character within a short, predetermined time window, and any strokes entered outside the predetermined time window are not included in the same recognition unit as other strokes provided during the time window. Such conventional systems are difficult to use and cause great frustration to users. Stroke-order-independent systems are free from these drawbacks and allow users to complete characters in any order and in any time frame as the user deems appropriate. Users can also make modifications to previously written characters (e.g., adding one or more strokes) after subsequently writing one or more characters in the handwriting input interface. In some embodiments, the user can also individually delete previously written characters (e.g., using the methods described later in connection with Figures 21A and 22B) and rewrite the characters in the same location within the handwriting input interface.

[0193] 10B and 10C, the second plurality of handwritten strokes spatially follows the first plurality of handwritten strokes along a default writing direction of the handwriting input interface of the user device (1028), and the second output character follows the first output character in a spatial arrangement along the default writing direction within the candidate display area of ​​the handwriting input interface. The user device receives a third handwritten stroke from the user to correct the first handwritten character (i.e., the handwritten character formed by the first plurality of handwritten strokes) (1030), the third handwritten stroke being received later in time than the first and second plurality of handwritten strokes. For example, the user may write two characters (e.g., " ) in a left-to-right spatial arrangement within the handwriting input area. The first plurality of strokes is the handwritten character " JPEG2025124752000054.jpg1011". The user actually uses the characters " Notice that I intended to write "JPEG2025124752000055.jpg1010", but left out one stroke. The second set of strokes is the handwritten " JPEG2025124752000056.jpg1010". JPEG2025124752000057.jpg1019" instead of " Then realizing that the user wanted to write "JPEG2025124752000058.jpg917", the user would simply type in the characters " Another vertical stroke can be written below the stroke for "JPEG2025124752000059.jpg1011", and the user device can then write that vertical stroke as a first recognition unit (e.g., " The user device assigns a new output character (e.g., " JPEG2025124752000061.jpg1011"), and this new output character is compared with the previous output character (e.g., " 10C , in response to receiving the third handwritten stroke, the user device assigns the third handwritten stroke to the same recognition unit as the first plurality of handwritten strokes based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes (1032). In some embodiments, the user device generates a corrected input image based on the first plurality of handwritten strokes and the third handwritten stroke (1034). The user device provides the corrected input image to a handwriting recognition model to perform real-time recognition of the corrected handwritten character (1036). In some embodiments, in response to receiving the third handwritten input, the user device displays a third output character corresponding to the corrected input image (1040), where the third output character replaces the first output character and is displayed simultaneously with the second output character in spatial alignment along the default writing direction.

[0194] In some embodiments, the handwriting recognition module recognizes handwriting input written in a default writing direction of left to right. For example, a user can write characters from left to right and in one or more lines. In response to this handwriting input, the handwriting input module optionally presents a recognition result including characters in a left-to-right spatial arrangement and in one or more lines. If the user selects a recognition result, the selected recognition result is entered into the text input area of ​​the user device. In some embodiments, the default writing direction is top to bottom. In some embodiments, the default writing direction is right to left. In some embodiments, the user optionally changes the default writing direction to an alternative writing direction after the recognition result is selected and the handwriting input area is cleared.

[0195] In some embodiments, the handwriting input module allows a user to input multi-character handwriting input into the handwriting input area and allows strokes to be deleted from the handwriting input one recognition unit at a time, rather than all recognition units at once. In some embodiments, the handwriting input module allows strokes to be deleted from the handwriting input one at a time. In some embodiments, the deletion of recognition units proceeds one by one in a direction opposite to the default writing direction, regardless of the order of the recognition units or strokes entered to create the current handwriting input. In some embodiments, the deletion of strokes proceeds one by one in the reverse order in which the strokes were entered within each recognition unit, and when all strokes within a recognition unit are deleted, the deletion of strokes proceeds to the next recognition unit in a direction opposite to the default writing direction.

[0196] In some embodiments, while the third output character and the second output character are simultaneously displayed as candidate recognition results in a candidate display area of ​​the handwriting input interface, the user device receives a delete input from the user, and in response to the delete input, the user device deletes the second output character from the recognition result while maintaining the third output character in the recognition result displayed in the candidate display area.

[0197] 10C , the user device renders the first plurality of handwritten strokes, the second plurality of handwritten strokes, and the third plurality of handwritten strokes in real time as each of the handwritten strokes is provided by the user (1042). In some embodiments, in response to receiving a delete input from the user, the user device deletes the rendering of each of the second plurality of handwritten inputs (e.g., corresponding to the second handwritten character) from the handwriting input area, while maintaining the rendering of each of the first plurality of handwritten strokes and the third handwritten strokes (e.g., corresponding together to the corrected first handwritten character) in the handwriting input area (1044). For example, if the user enters the character sequence " If the user enters a delete input after providing the missing vertical stroke in "JPEG2025124752000063.jpg917", the character " The strokes within the recognition unit for "JPEG2025124752000064.jpg1010" are removed from the handwriting input area, and the character " JPEG2025124752000065.jpg1010" is the recognition result " After this deletion, the characters " The strokes for "JPEG2025124752000067.jpg1010" are kept within the handwriting input area, and the recognition result is the character " Only "JPEG2025124752000068.jpg1010" is displayed.

[0198] In some embodiments, the handwritten characters are multi-stroke Chinese characters. In some embodiments, the first plurality of handwritten inputs are provided in a cursive typeface. In some embodiments, the first plurality of handwritten inputs are provided in a cursive typeface, and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwritten characters are written in cursive Arabic. In some embodiments, the handwritten characters are written in cursive other script.

[0199] In some embodiments, the user device establishes respective predefined constraints for a set of acceptable dimensions for handwritten character input, divides the currently accumulated handwritten strokes into a plurality of recognition units based on the respective predefined constraints, and generates respective input images from each of the recognition units, which are provided to the handwriting recognition model and recognized as corresponding output characters.

[0200] In some embodiments, the user device receives additional handwritten strokes from the user after dividing the currently stored number of handwritten strokes, and assigns the additional handwritten strokes to respective ones of the number of recognition units based on spatial locations of the additional handwritten strokes relative to the number of recognition units.

[0201] Attention is now directed to an exemplary user interface for providing handwriting recognition and input on a user device. In some embodiments, this exemplary user interface is provided on a user device based on a multi-script handwriting recognition model that provides real-time, stroke-order-independent handwriting recognition of a user's handwritten input. In some embodiments, this exemplary user interface is that of exemplary handwriting input interface 802 (e.g., as shown in FIGS. 8A and 8B ), which includes handwriting input area 804, candidate display area 804, and text input area 808. In some embodiments, exemplary handwriting input interface 802 also includes multiple control elements 1102, such as a delete button, a space bar, an enter button, a keyboard toggle button, etc. One or more other areas and / or elements may be provided within handwriting input interface 802 to enable additional functionality, as described below.

[0202] As described herein, a multi-script handwriting recognition model can have an extremely broad repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwritten input, the recognition model is very likely to identify a large number of output characters, all of which have a reasonably good likelihood of being the characters intended by the user. On user devices with limited display area, it is advantageous to initially provide only a subset of the recognition results, while keeping other results available upon user request.

[0203] 11A-11G show exemplary user interfaces for displaying a subset of recognition results in a normal view of the candidate display area, along with affordances for invoking an expanded view of the candidate display area to display the remainder of the recognition results. Furthermore, in the expanded view of the candidate display area, the recognition results are categorized into different categories and displayed on different tabbed pages within the expanded view.

[0204] 11A shows an exemplary handwriting input interface 802. The handwriting input interface includes a handwriting input area 804, a candidate display area 806, and a text input area 808. One or more control elements 1102 are also included within the handwriting input interface 1002.

[0205] As shown in FIG. 11A, the candidate display area 806 optionally includes an area for displaying one or more recognition results and an affordance 1104 (e.g., an expansion icon) for invoking an expanded version of the candidate display area 806.

[0206] 11A-11C show that when a user provides one or more handwritten strokes (e.g., strokes 1106, 1108, and 1110) in the handwriting input area 804, the user device identifies and displays a set of respective recognition results corresponding to the strokes currently stored in the handwriting input area 804. As shown in FIG. 11B, after the user inputs the first stroke 1106, the user device identifies and displays three recognition results 1112, 1114, and 1116 (e.g., the characters " / ", "1", and ","). In some embodiments, a small number of candidate characters are displayed in the candidate display area 806 in order according to the recognition confidence associated with each character.

[0207] In some embodiments, the top-ranked candidate result (e.g., " / ") is provisionally displayed in the text entry area 808, for example, in box 1118. The user can optionally confirm that the top-ranked candidate is the intended input with a simple confirmation (e.g., by pressing the "Enter" key or providing a double-tap gesture in the writing entry area).

[0208] 11C shows that if the user enters two more strokes 1108 and 1110 into the handwriting input area 804 before selecting one of the candidate recognition results, those additional strokes are rendered in the handwriting input area 804 along with the initial stroke 1106, and the candidate results are updated to reflect those changes to the recognition unit identified from the currently accumulated handwriting input. As shown in FIG. 11C , based on the three strokes, the user device identifies a single recognition unit. Based on this single recognition unit, the user device identifies and displays several recognition results 1118-1124. In some embodiments, one or more of these recognition results (e.g., 1118 and 1122) currently displayed in the candidate display area 806 are respective representative candidate characters selected from among multiple similar-looking candidate characters for the current handwriting input.

[0209] 11C and 11D, when a user selects affordance 1104 (e.g., using a tap gesture with contact 1126 on affordance 1104), the candidate display area changes from a normal view (e.g., as shown in FIG. 11C) to an expanded view (e.g., as shown in FIG. 11D). In some embodiments, this expanded view shows all of the recognition results (e.g., candidate characters) that have been identified for the current handwriting input.

[0210] In some embodiments, the initially displayed normal view of the candidate display area 806 shows only the most commonly used characters used in the respective script or language, while the expanded view shows all candidate characters, including less frequently used characters in the script or language. This expanded view of the candidate display area can be designed in various ways. Figures 11D-11G show example designs of the expanded candidate display area, according to some embodiments.

[0211] 11D, in some embodiments, expanded candidate display area 1128 includes one or more tabbed pages (e.g., pages 1130, 1132, 1134, and 1136), each presenting a respective category of candidate characters. The tabbed design shown in FIG. 11D allows a user to quickly find a desired category of characters and then find the character the user intended to input within that corresponding tabbed page.

[0212] 11D, a first tabbed page 1130 displays all candidate characters that have been identified for the currently stored handwritten input, including both commonly used characters and less frequently used characters. As shown in FIG. 11D, the tabbed page 1130 displays all of the characters that were displayed in the initial candidate display area 806 in FIG. 11C, as well as some additional characters that were not included in the initial candidate display area 806 (e.g., " JPEG2025124752000069.jpg1012", "β", " JPEG2025124752000070.jpg108" and other files.

[0213] In some embodiments, the characters displayed in the initial candidate display area 806 include only characters from the set of commonly used characters associated with a particular script (e.g., all characters in the basic block of the CJK script as encoded according to the Unicode standard). In some embodiments, the characters displayed in the expanded candidate display area 1128 also include a set of less frequent characters associated with that script (e.g., all characters in the extended block of the CJK script as encoded according to the Unicode standard). In some embodiments, the expanded candidate display area 1128 also includes candidate characters from other scripts that are not commonly used by users, such as Greek, Arabic, and / or Emoji scripts.

[0214] In some embodiments, as shown in FIG. 11D, expanded candidate display area 1128 includes respective tabbed pages 1130, 1132, 1134, and 1138, each corresponding to a respective candidate character category (e.g., all characters, low-frequency characters, characters from Latin scripts, and characters from Emoji scripts, respectively). FIGS. 11E-11G illustrate that a user can select each of the various tabbed pages to reveal candidate characters within the corresponding category. FIG. 11E shows only low-frequency characters (e.g., characters from an extended block of CJK scripts) that correspond to the current handwritten input. FIG. 11F shows only Latin and Greek characters that correspond to the current handwritten input. FIG. 11G shows only Emoji characters that correspond to the current handwritten input.

[0215] In some embodiments, the expanded candidate display area 1128 further includes one or more affordances for sorting the candidate characters within each tabbed page based on respective criteria (e.g., based on phonetic spelling, based on number of strokes, based on radicals, etc.) The ability to sort the candidate characters within each category according to criteria other than recognition confidence score provides the user with additional capabilities to quickly find desired candidate characters for text input.

[0216] 11H-11K show that in some embodiments, similar-looking candidate characters can be grouped, and only a representative character from each group of similar-looking candidate characters is presented in the initial candidate display area 806. Because the multi-script recognition model described herein can produce many candidate characters that are approximately equally valid for a given handwriting input, the recognition model is not necessarily capable of eliminating one candidate at the expense of another similar-looking candidate. On devices with limited display areas, displaying many similar-looking candidates all at once does not help the user pick the correct character because subtle distinctions are not easily visible, and even if the user can see the desired character, it can be difficult to select it using a finger or stylus from a very crowded display.

[0217] In some embodiments, to address the above problem, the user device identifies candidate characters that have a high similarity to each other (e.g., according to a concordance or dictionary of similar-looking characters, or some image-based criteria) and groups the candidate characters into respective groups. In some embodiments, one or more groups of similar-looking characters may be identified from a set of candidate characters for a given handwritten input. In some embodiments, the user device identifies a representative candidate character from among multiple similar-looking candidate characters in the same group, and displays only the representative candidate in the initial candidate display area 806. If a commonly used character does not look sufficiently similar to any of the other candidate characters, the character itself is displayed. In some embodiments, as shown in FIG. 11H, the representative candidate character for each group (e.g., candidate character 1118 " JPEG2025124752000071.jpg1010" and 1122 "T") are candidate characters that do not belong to any group (for example, candidate character 1120 " JPEG2025124752000072.jpg910" and 1124"J"). In some embodiments, the criteria for selecting a representative character for a group are based on the relative frequency of use of the candidate characters within that group. In some embodiments, other criteria can be used.

[0218] In some embodiments, once the representative character is displayed to the user, the user can optionally expand the candidate display area 806 to show similar-looking candidate characters in the expanded view. In some embodiments, selection of a particular representative character can result in an expanded view of only the candidate characters in the same group as the selected representative character.

[0219] Various designs for providing an enlarged view of similar-looking candidates are possible. Figures 11H-11K show one embodiment in which a predefined gesture (e.g., a magnify gesture) detected over a representative candidate character (e.g., representative character 1118) invokes an enlarged view of the representative candidate character. The predefined gesture (e.g., a magnify gesture) for invoking the enlarged view is different from the predefined gesture (e.g., a tap gesture) for selecting a representative character for text input.

[0220] As shown in FIGS. 11H and 11I, when a user provides a magnification gesture (e.g., as indicated by two contacts 1138 and 1140 moving away from each other) over the first representative character 1118, the area displaying the representative character 1118 expands and three similar-looking candidate characters (e.g., " JPEG2025124752000073.jpg1010", " JPEG2025124752000074.jpg1012" and " JPEG2025124752000075.jpg108") is not included in the same magnification group as other candidate characters (e.g., " 1142, 1144, and 1146, respectively).

[0221] As shown in Figure 11I, when presented in a magnified view, three similar-looking candidate characters (e.g., “ JPEG2025124752000077.jpg1010", " JPEG2025124752000078.jpg1012" and " 11J and 11K, the user can select the second character (e.g., " ) shown in box 1144 in the enlarged view. JPEG2025124752000080.jpg1012") is selected (by touch 1148). In response, the selected character (e.g., " JPEG2025124752000081.jpg1012") is entered into the text entry area 808 with the insertion point indicated by the cursor. As shown in FIG. 11K, once the character is selected, the handwritten input in the handwriting entry area 804 and the candidate character in the candidate display area 806 (or an enlarged view of the candidate display area) are cleared for subsequent handwritten input.

[0222] In some embodiments, if the user does not see the desired candidate character in the enlarged view of the first representative candidate character 1142, the user can optionally use the same gesture to enlarge other characters displayed in the candidate display area 806. In some embodiments, upon enlarging another representative character in the candidate display area 806, the currently presented enlarged view is automatically restored to the normal view. In some embodiments, the user optionally uses a zoom-out gesture to restore the currently enlarged view to the normal view. In some embodiments, the user can scroll the candidate display area 806 (e.g., left or right) to reveal other candidate characters that are not visible in the candidate display area 806.

[0223] 12A and 12B are flow diagrams of an example process 1200 in which a first subset of recognition results is presented in an initial candidate display area, while a second subset of recognition results is presented in an expanded candidate display area that is hidden and not visible until specifically invoked by the user. In example process 1200, a device identifies a subset of recognition results from a plurality of handwriting recognition results for a handwritten input that have a visual similarity level that exceeds a predetermined threshold. The user device then selects a representative recognition result from the subset of recognition results and displays the selected representative recognition result in the candidate display area of ​​the display. Process 1200 is shown in FIGS. 11A-11K.

[0224] As shown in FIG. 12A , in exemplary process 1200, a user device receives handwritten input from a user (1202). The handwritten input includes one or more handwritten strokes (e.g., 1106, 1108, 1110 in FIG. 11C ) provided within a handwriting input area (e.g., 806 in FIG. 11C ) of a handwriting input interface (e.g., 802 in FIG. 11C ). The user device identifies (1204) a plurality of output characters (e.g., characters shown within tabbed page 1130 in FIG. 11C ) for the handwritten input based on a handwriting recognition model. The user device classifies (1206) the plurality of output characters into two or more categories based on predefined categorization criteria. In some embodiments, the predefined categorization criteria determine (1208) whether each character is a commonly used character or a low-frequency character.

[0225] In some embodiments, the user device displays (1210) each output character in a first category (e.g., commonly used characters) of two or more categories in an initial view of a candidate display area (e.g., 806 as shown in FIG. 11C) of the handwriting input interface, and this initial view of the candidate display area is provided simultaneously with an affordance (e.g., 1104 in FIG. 11C) for invoking an expanded view of the candidate display area (e.g., 1128 in FIG. 11D).

[0226] In some embodiments, the user device receives 1212 a user input selecting an affordance for invoking an expanded view, e.g., as shown in Figure 11C. In response to the user input, the user device displays 1214 in the expanded view of the candidate display area each output character in the first category and each output character in at least a second category of the two or more categories that was not previously displayed in the initial view of the candidate display area, e.g., as shown in Figure 11D.

[0227] In some embodiments, each character in the first category is a character found in a dictionary of commonly used characters and each character in the second category is a character found in a dictionary of infrequent characters, In some embodiments, the dictionary of commonly used characters and the dictionary of infrequent characters are dynamically adjusted or updated based on usage history associated with the user device.

[0228] In some embodiments, the user device identifies (1216) from the plurality of output characters groups of characters that are visually similar to one another according to a predefined similarity criterion (e.g., based on a dictionary of similar characters or based on some spatially derived feature). In some embodiments, the user device selects a representative character from the group of visually similar characters based on a predefined selection criterion (e.g., based on historical frequency of use). In some embodiments, the predefined selection criterion is based on the relative frequency of use of the character within the group. In some embodiments, the predefined selection criterion is based on a preferred input language associated with the device. In some embodiments, the representative candidates are based on other factors that indicate the likelihood that each candidate is the user's intended input. These factors include, for example, whether the candidate character belongs to a script for a soft keyboard currently installed on the user's device, or whether the candidate character is among the set of most commonly used characters for a particular language associated with the user or the user device.

[0229] In some embodiments, the user device displays a representative character (e.g., " ) in the initial view of the candidate display area (e.g., 806 in FIG. 11H ). JPEG2025124752000082.jpg1010") compared to other characters in that visually similar group of characters (e.g., " JPEG2025124752000083.jpg1012", " 11H )。 In some embodiments, a visual indication (e.g., selective visual highlighting, special background) is provided in the initial view of the candidate display area to indicate whether each candidate character is a representative character of the group or a regular candidate character that is not in any group. In some embodiments, the user device receives (1222) from the user a predefined magnification input (e.g., a magnification gesture) directed to the representative character displayed in the initial view of the candidate display area, as shown in FIG. 11H. In some embodiments, in response to receiving this predefined magnification input, the user device simultaneously displays (1224) a magnified view of the representative character and a respective magnified view of one or more other characters in the group of visually similar characters, as shown in FIG. 11I.

[0230] In some embodiments, the default magnification input is a magnification gesture detected over a representative character displayed in the candidate display area. In some embodiments, the default magnification input is a contact detected over a representative character displayed in the candidate display area and sustained for longer than a default threshold time. In some embodiments, the sustained contact for expanding the group has a longer threshold duration than a tap gesture for selecting a representative character for text input.

[0231] In some embodiments, each representative character is simultaneously displayed with a corresponding affordance (e.g., a corresponding magnification button) for invoking a magnified view of its group of similar-looking candidate characters. In some embodiments, the default magnification input is selection of the corresponding affordance associated with the representative character.

[0232] As described herein, in some embodiments, the repertoire of multi-script handwriting recognition models includes emoji scripts. The handwriting input recognition module can recognize emoji based on a user's handwritten input. In some embodiments, the handwriting recognition module presents both an emoji identified directly from the handwriting and a character or word in a natural human language that represents the identified emoji. In some embodiments, the handwriting input module recognizes a character or word in a natural human language based on the user's handwritten input and presents both the recognized character or word and an emoji corresponding to the recognized character or word. In other words, the handwriting input module provides a method for inputting emoji without switching from a handwriting input interface to an emoji keyboard. Furthermore, the handwriting input module also provides a method for inputting regular natural language characters by drawing emoji by hand. FIGS. 13A-13E provide exemplary user interfaces illustrating these various methods of inputting emoji and regular natural language characters.

[0233] 13A shows an exemplary handwriting input interface 802 invoked under a chat application. The handwriting input interface 802 includes a handwriting input area 804, a candidate display area 806, and a text entry area 808. In some embodiments, once a user is satisfied with the text composition in the text entry area 808, the user can select to send the text composition to another participant in the current chat session. The dialogue history of the chat session is shown in a dialogue panel 1302. In this example, the user can select a chat message 1304 (e.g., "Happy Birthday") displayed in the dialogue panel 1302. JPEG2025124752000085.jpg1110" is received.

[0234] 13B, a user provides handwritten input 1306 for the English word "Thanks" in handwritten input area 804. In response to handwritten input 1306, the user device identifies several candidate recognition results (e.g., recognition results 1308, 1310, and 1312). The top-ranked recognition result 1303 has been provisionally entered in box 1314 in text input area 808.

[0235] As shown in FIG. 13C , after the user inputs the handwritten word “Thanks” in the handwriting input area 806, the user then draws a stylized exclamation point with stroke 1316 (e.g., an elongated circle with a round circle below) in the handwriting input area 806. The user device recognizes that this additional stroke 1316 forms a recognition unit separate from other recognition units previously recognized from the accumulated handwritten strokes 1306 in the handwriting input area 806. Based on this newly input recognition unit (i.e., the recognition unit formed by stroke 1316), the user device uses a handwriting recognition model to identify an emoji (e.g., a stylized “!”). Based on this recognized emoji, the user device presents a first recognition result 1318 (e.g., “Thanks!” with a stylized “!”) in the candidate display area 806. Furthermore, the user device also identifies the number “8,” which is also visually similar to the newly input recognition unit. Based on this recognized number, the user device presents a second recognition result 1322 (e.g., "Thanks 8") in the candidate display area 806. Furthermore, based on the identified emoji (e.g., the stylized "!"), the user device also identifies a regular character corresponding to the emoji (e.g., the regular character "!"). Based on this indirectly recognized regular character, the user device presents a third recognition result 1320 (e.g., "Thanks!" with the regular "!") in the candidate display area 806. At this point, the user can select any one of the candidate recognition results 1318, 1320, and 1322 and enter the candidate recognition result in the text input area 808.

[0236] As shown in FIG. 13D , the user continues to provide additional handwritten strokes 1324 within the handwriting input area 806. This time, the user is drawing a stylized exclamation point followed by a heart symbol. In response to this new handwritten stroke 1324, the user device recognizes that the newly provided handwritten stroke 1324 forms yet another new recognition unit. Based on this new recognition unit, the user device selects the emoji " JPEG2025124752000086.jpg99", and alternatively the digit "0". Based on these new candidate characters recognized from the new recognition units, the user device generates two updated candidate recognition results 1326 and 1330 (e.g., "Thanks" In some embodiments, the user device presents the identified emoji (e.g., " 13D , the user device further identifies a regular character or word (e.g., “Love”) corresponding to the emoji “ JPEG2025124752000088.jpg99”. Based on the identified regular character or word for the recognized emoji, the user device presents a third recognition result 1328 in which the recognized emoji is replaced with the corresponding regular character or word. As shown in FIG. 13D , in the recognition result 1328, the emoji “ JPEG2025124752000089.jpg96" is replaced with the usual exclamation mark "!" and the emoji " JPEG2025124752000090.jpg99" has been replaced with the regular characters or word "Love".

[0237] As shown in FIG. 13E, the user may select one of the candidate recognition results (e.g., the mixed-script text "Thanks" The user selects a candidate result 1326 showing "JPEG2025124752000091.jpg812" and the text of this selected recognition result is entered into the text entry area 808 and then sent to other participants in the chat session. A message bubble 1332 shows the text of the message in the dialogue panel 1302.

[0238] 14 is a flow diagram of an example process 1400 in which a user inputs emojis using handwriting input. Figures 13A-13E illustrate an example process 1400, according to some embodiments.

[0239] In process 1400, a user device receives handwritten input from a user (1402). The handwritten input includes a plurality of handwritten strokes provided within a handwriting input area of ​​a handwriting input interface. In some embodiments, the user device recognizes a plurality of output characters from the handwritten input based on a handwriting recognition model (1404). In some embodiments, the output characters include at least a first emoji (e.g., the stylized exclamation point " JPEG2025124752000092.jpg96" or emoji " JPEG2025124752000093.jpg99") and at least a first character from a script of a natural human language (e.g., a character from the word "Thanks" in FIG. 13D). In some embodiments, the user device may, for example, as shown in FIG. 13D, include those first emoji (e.g., a stylized exclamation point " JPEG2025124752000094.jpg96" or emoji " The recognition result (e.g., result 1326 in FIG. 13D ) including the first character from the script of the natural human language (e.g., the character from the word “Thanks” in FIG. 13D ) is displayed in the candidate display area of ​​the handwriting input interface (1406).

[0240] In some embodiments, based on the handwriting recognition model, the user device optionally recognizes (1408) from the handwritten input at least a first semantic unit (e.g., the word "Thanks"), the first semantic unit including respective characters, words, or phrases capable of conveying respective semantic meanings in respective human languages. In some embodiments, the user device identifies (1410) a second emoji (e.g., a "handshake" emoji) associated with the first semantic unit (e.g., the word "Thanks") recognized from the handwritten input. In some embodiments, the user device displays (1411) a second recognition result (e.g., a "handshake" emoji followed by ")" in a candidate display area of ​​the handwriting input interface, the second recognition result including at least the second emoji identified from the first semantic unit (e.g., the word "Thanks"). JPEG2025124752000096.jpg96" and " In some embodiments, displaying the second recognition result includes displaying 1412 a third recognition result (e.g., a recognition result showing an emoji for "JPEG2025124752000097.jpg99") that includes at least the first semantic unit (e.g., the word "Thanks"). The method further includes displaying the second recognition result simultaneously with the original image ("JPEG2025124752000098.jpg812").

[0241] In some embodiments, a user receives a user input selecting a first recognition result displayed in the candidate display area. In some embodiments, in response to the user input, the user device enters text of the selected first recognition result into a text input area of ​​the handwriting input interface, the text including at least a first emoji and a first character from a script of a natural human language. In other words, a user can enter a mixed-script text input using a single handwriting input (albeit a handwriting input including multiple strokes) in the handwriting input area without switching between a natural language keyboard and an emoji keyboard.

[0242] In some embodiments, the handwriting recognition model is trained on a multi-script training corpus that includes writing samples corresponding to characters from at least three non-overlapping scripts, the three non-overlapping scripts including a set of emoji, Chinese, and Latin scripts.

[0243] In some embodiments, the user device may recognize a first emoji (e.g., " In some embodiments, the user device identifies (1414) a second semantic unit (e.g., the word "Love") that corresponds to the first emoji (e.g., the emoji in "JPEG2025124752000099.jpg99") in a candidate display area of ​​the handwriting input interface. In some embodiments, the user device displays (1416) a fourth recognition result (e.g., 1328 in FIG. 13D ) that includes at least the second semantic unit (e.g., the word “Love”) identified from the emoji in “JPEG2025124752000100.jpg99.” In some embodiments, the user device displays (1416) the first recognition result (e.g., the result “Thanks” in the candidate display area) in the candidate display area, as shown in FIG. JPEG2025124752000101.jpg812") and the fourth recognition result (e.g., result 1328 "Thanks! Love").

[0244] In some embodiments, the user device allows the user to input regular text by drawing emoji. For example, if the user does not know how to spell the word "elephant," the user optionally draws a stylized emoji for "elephant" in the handwriting input area, and if the user device can correctly recognize the handwriting input as an emoji for "elephant," the user device also optionally presents the word "elephant" in regular text as one of the recognition results displayed in the candidate display area. In another example, the user may draw the Chinese character " Instead of writing "JPEG2025124752000102.jpg1010", the user device may draw a stylized cat. If the user device identifies an emoji for "cat" based on the handwriting input provided by the user, the user device may also optionally display the Chinese character " , which means "cat" in Chinese, in the candidate display area along with the emoji for "cat". JPEG2025124752000103.jpg1010." By presenting regular text for recognized emoji, the user device provides an alternative way of entering complex characters or words using a small number of stylized strokes commonly associated with known emoji. In some embodiments, the user device stores a dictionary that associates emoji with their corresponding regular text (e.g., characters, words, phrases, symbols, etc.) in one or more preferred scripts or languages ​​(e.g., English or Chinese).

[0245] In some embodiments, the user device recognizes emoji based on the visual similarity of the emoji to images generated from the handwritten input. In some embodiments, to enable recognition of emoji from handwritten input, a handwriting recognition model used on the user device is trained using a training corpus that includes both handwriting samples corresponding to characters of scripts of natural human languages ​​and handwriting samples corresponding to sets of artificially designed emoji. In some embodiments, emoji related to the same semantic concept may have different appearances when used in mixed input with text in different natural languages. For example, an emoji related to the semantic concept of "Love" may be a "heart" emoji when presented in regular text in one natural language (e.g., Japanese) and a "kiss" emoji when presented in regular text in another natural language (e.g., English or French).

[0246] As described herein, when performing recognition of multi-character handwriting input, the handwriting input module performs segmentation of the handwriting input currently accumulated in the handwriting input area and separates the accumulated strokes into one or more recognition units. One of the parameters used to determine how to segment the handwriting input can be how the strokes are clustered in the handwriting input area and the distance between different stroke clusters. People have different writing styles. Some people tend to write very sparsely, with large distances between strokes or between different parts of the same character, while other people tend to write very densely, with very small distances between strokes or between different characters. Even for the same user, imperfect planning can cause handwriting to deviate from a balanced appearance and become skewed, stretched, or compressed in various ways. As described herein, the multi-script handwriting recognition model provides stroke-order-independent recognition, thereby allowing users to write characters or parts of characters out of order. As a result, achieving spatial uniformity and balance in handwriting input between characters can be difficult.

[0247] In some embodiments, the handwriting input model described herein provides a way for a user to inform the handwriting input module whether to merge two adjacent recognition units into a single recognition unit or separate a single recognition unit into two separate recognition units, so that the handwriting input module can correct its initial segmentation and produce the result intended by the user.

[0248] 15A-15J illustrate some example user interfaces and processes in which a user provides predefined pinch and spread gestures to modify recognition units identified by a user device.

[0249] 15A and 15B, a user inputs multiple handwritten strokes 1502 (e.g., three strokes) into the handwriting input area 806 of the handwriting input interface 802. The user device identifies a single recognition unit based on the currently stored handwritten strokes 1502, and displays three candidate characters 1508 (e.g., " ... JPEG2025124752000104.jpg108", " JPEG2025124752000105.jpg109" and " They present the following image: "JPEG2025124752000106.jpg109".

[0250] 15C shows that the user has further input a few additional strokes 1510 to the right of the initial handwritten stroke 1502 within the handwriting input area 606. The user device determines (e.g., based on the dimensions and spatial distribution of the strokes 1502 and 1510) that stroke 1502 and stroke 1510 should be considered as two separate recognition units. Based on this separation of the recognition units, the user device provides input images of the first and second recognition units to a handwriting recognition model to obtain two sets of candidate characters. The user device then generates multiple recognition results (e.g., 1512, 1514, 1516, and 1518) based on various combinations of the recognition characters. Each recognition result includes a recognized character for the first recognition unit and a recognized character for the second recognition unit. As shown in FIG. 15C, the multiple recognition results 1512, 1514, 1516, and 1518 each include two recognized characters.

[0251] In this example, the user actually intended for this handwritten input to be recognized as a single character, but the handwritten character (e.g., " JPEG2025124752000107.jpg1010"), the left part (for example, the left radical " JPEG2025124752000108.jpg108") and the right part (e.g., the right radical " Suppose a user inadvertently leaves an excessively large space between the current handwritten input and the next character (e.g., "JPEG2025124752000109.jpg98"). By viewing the results presented in candidate display area 806 (e.g., 1512, 1514, 1516, and 1518), the user realizes that the user device has incorrectly divided the current handwritten input into two recognition units. Although this division may be based on objective criteria, the user does not want to delete the current handwritten input and rewrite the entire character with a smaller distance between the left and right portions.

[0252] 15D, the user uses a pinch gesture over two clusters of handwritten strokes 1502 and 1510 to indicate to the handwriting input module that the two recognition units identified by the handwriting input module should be merged into a single recognition unit. This pinch gesture is indicated by two contacts 1520 and 1522 on the touch-sensitive surface moving toward each other.

[0253] 15E shows that, in response to the user's pinch gesture, the user device corrects the segmentation of the currently stored handwriting input (e.g., strokes 1502 and 1510) and fuses the handwriting strokes into a single recognition unit. As shown in FIG. 15E, the user device provides an input image based on the corrected recognition unit to a handwriting recognition model, which generates three new candidate characters 1524, 1526, and 1528 (e.g., " ) for the corrected recognition unit. JPEG2025124752000110.jpg1010", " JPEG2025124752000111.jpg1010" and " 15E , the user device optionally adjusts the rendering of the handwritten input in handwriting input area 806 so that the distance between the left and right clusters of handwritten strokes is reduced. In some embodiments, the user device does not change the rendering of the handwritten input shown in handwriting input area 608 in response to the pinch gesture. In some embodiments, the user device distinguishes the pinch gesture from the input strokes based on two simultaneous contacts (as opposed to a single contact) detected in handwriting input area 806.

[0254] As shown in FIG. 15F, the user may select previously entered handwritten input (i.e., the characters " JPEG2025124752000113.jpg1010) to the right of the strokes related to "JPEG2025124752000113.jpg1010." The user device determines that the newly input strokes 1530 are new recognition units, and selects candidate characters (e.g., " The user device then recognizes the newly identified character (e.g., " JPEG2025124752000115.jpg99”) with the candidate characters for the previously identified recognition unit and presents several different recognition results (e.g., results 1532 and 1534) in candidate display area 806.

[0255] Following handwritten stroke 1530, the user continues to write additional strokes 1536 (e.g., three additional strokes) to the right of stroke 1530, as shown in Figure 15G. Because the horizontal distance between stroke 1530 and stroke 1536 is extremely small, the user device determines that stroke 1530 and stroke 1536 belong to the same recognition unit and provides the input image formed by strokes 1530 and 1536 to the handwriting recognition model. The handwriting recognition model identifies three different candidate characters for the corrected recognition unit and generates two corrected recognition results 1538 and 1540 for the currently stored handwritten input.

[0256] In this example, the last two sets of strokes 1530 and 1536 are actually two separate characters (e.g., " Assume that the set of two strokes 1530 and 1536 is intended as ".jpg" ("JPEG2025124752000116.jpg99") and "±" ("±"). After the user sees that the user device has mistakenly combined the set of two strokes 1530 and 1536 into a single recognition unit, the user subsequently provides a magnification gesture to inform the user device that the set of two strokes 1530 and 1536 should be separated into two separate recognition units. As shown in FIG. 15H, the user places two contacts 1542 and 1544 around strokes 1530 and 1536 and then moves the two contacts away from each other in a generally horizontal direction (i.e., along the default writing direction).

[0257] 15I shows that in response to the user's expansion gesture, the user device corrects the previous division of the currently stored handwritten input and assigns stroke 1530 and stroke 1536 into two contiguous recognition units. Based on the input image generated for those two separate recognition units, the user device identifies one or more candidate characters for the first recognition unit based on stroke 1530 and one or more candidate characters for the second recognition unit based on stroke 1536. The user device then generates two new recognition results 1546 and 1548 based on various combinations of those recognition characters. In some embodiments, the user device optionally modifies the rendering of strokes 1536 and 1536 to reflect the separation of the previously identified recognition units.

[0258] 15J-15K, the user has selected (as indicated by contact 1550) one of the candidate recognition results displayed in candidate display area 806, and the selected recognition result (e.g., result 1548) is entered into the user interface's text entry area 808. After the selected recognition result is entered into text entry area 808, both candidate display area 806 and handwriting entry area 804 are cleared, ready to display subsequent user input.

[0259] 16A-16B are flow diagrams of an example process 1600 for informing a user how to split or correct existing splits in a current handwriting input using predefined gestures (e.g., pinch and / or spread gestures). Figures 15J and 15K provide illustrations of an example process 1600, according to some embodiments.

[0260] In some embodiments, a user device receives handwriting input from a user (1602). The handwriting input includes a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwritten strokes in real time (1604) in a handwriting input area of ​​a handwriting input interface (e.g., handwriting input area 806 in FIGS. 15A-15K). The user device receives one of a pinch gesture input and a spread gesture input over the plurality of handwritten strokes, as shown, for example, in FIGS. 15D and 15H.

[0261] In some embodiments, upon receiving the pinch gesture input, the user device generates (1606) a first recognition result based on the multiple handwritten strokes by processing the multiple handwritten strokes as a single recognition unit, for example, as shown in FIGS. 15C-15E.

[0262] In some embodiments, upon receiving the magnification gesture input, the user device generates (1608) a second recognition result based on the multiple handwritten strokes by treating the multiple handwritten strokes as two separate recognition units separated by the magnification gesture input, for example, as shown in Figures 15G-15I.

[0263] In some embodiments, upon generating one of the first recognition result and the second recognition result, the user device displays the generated recognition result in a candidate display area of ​​the handwriting input interface, as shown, for example, in Figures 15E and 15I.

[0264] In some embodiments, the pinch gesture input comprises two simultaneous contacts on the touch-sensitive surface that converge towards each other within an area occupied by multiple handwritten strokes, hi some embodiments, the spread gesture input comprises two simultaneous contacts on the touch-sensitive surface that diverge from each other within an area occupied by multiple handwritten strokes.

[0265] In some embodiments, the user device identifies two adjacent recognition units from the plurality of handwritten strokes (e.g., 1614). The user device displays (1616) initial recognition results (e.g., results 1512, 1514, 1516, and 1518 in FIG. 15C ) including each character recognized from the two adjacent recognition units in a candidate display area, for example, as shown in FIG. 15C . In some embodiments, when displaying a first recognition result (e.g., results 1524, 1526, or 1528 in FIG. 15E ) in response to a pinch gesture, the user device replaces (1618) the initial recognition result with the first recognition result in the candidate display area. In some embodiments, the user device receives (1620) a pinch gesture input while the initial recognition result is displayed in the candidate display area, as shown in FIG. 15D . In some embodiments, in response to this pinch gesture input, the user device re-renders (1622) the handwritten strokes in the handwriting input area to reduce the distance between two adjacent recognition units, for example, as shown in FIG. 15E.

[0266] In some embodiments, the user device identifies (1624) a single recognition unit from the multiple handwritten strokes. The user device displays the character recognized from the single recognition unit (e.g., " JPEG2025124752000117.jpg99'' 15G ) (1626). In some embodiments, if a second recognition result (e.g., result 1546 or 1548 in FIG. 15I ) is to be displayed in response to the zoom-in gesture, the user device replaces (1628) the initial recognition result (e.g., result 1538 or 1540) with the second recognition result (e.g., result 1546 or 1548) in the candidate display area, as shown in, for example, FIGS. 15H, 15I . In some embodiments, the user device receives (1630) a zoom-in gesture input while the initial recognition result is displayed in the candidate display area, as shown in FIG. 15H . In some embodiments, in response to this magnification gesture input, the user device re-renders the handwritten strokes (1632) to increase the distance between the first subset of strokes assigned to the first recognition unit and the second subset of handwritten strokes assigned to the second recognition unit within the handwritten input area, as shown in Figures 15H and 15I.

[0267] In some embodiments, immediately after a user provides strokes and realizes that the strokes may be too spread out for correct division based on a standard segmentation process, the user optionally provides a pinch gesture to signal the user device to treat the multiple strokes as a single recognition unit. The user device can distinguish the pinch gesture from regular strokes based on the two simultaneous contacts present in the pinch gesture. Similarly, in some embodiments, immediately after a user provides strokes and realizes that the strokes may be too close together for correct division based on a standard segmentation process, the user optionally provides a spread gesture to signal the user device to treat the multiple strokes as two separate recognition units. The user device can distinguish the spread gesture from regular strokes based on the two simultaneous contacts present in the pinch gesture.

[0268] In some embodiments, the direction of movement of a pinch or spread gesture is optionally used to provide further guidance on how to split the stroke under the gesture. For example, if multi-line handwriting input is enabled for the handwriting input area, a pinch gesture with two contacts moving vertically can inform the handwriting input module to merge two recognition units identified in two adjacent lines into a single recognition unit (e.g., as an upper radical and a lower radical). Similarly, a spread gesture with two contacts moving vertically can inform the handwriting input module to separate a single recognition unit into two recognition units in two adjacent lines. In some embodiments, pinch and spread gestures also provide guidance on splitting sub-portions of character input, for example, merging two sub-components or separating a compound character ( Guidance can also be provided for separating a single component within a single image (e.g., JPEG2025124752000119.jpg1070) into various portions of that compound character (e.g., the top, bottom, left, or right portion). This is particularly useful for recognizing complex compound Chinese characters, because users tend to lose the correct proportions and balance when writing complex compound characters by hand. Being able to adjust the proportions and balance of the handwritten input after the handwritten input is completed, for example, by pinch and spread gestures, is particularly useful for users to input the correct character without having to make several attempts to arrive at the correct proportions and balance.

[0269] As described herein, the handwriting input module allows a user to input multi-character handwritten input and allows out-of-sequence strokes for the multi-character handwritten input within the handwriting input area, within a character, across multiple characters, or even across multiple words, sentences, and / or lines. In some embodiments, the handwriting input module also provides character-by-character deletion within the handwriting input area, where the order of character deletion is in the reverse direction of writing and is independent of when the strokes for each character were provided in the handwriting input area. In some embodiments, deletion of each recognition unit (e.g., character or radical) within the handwriting input area is optionally performed stroke-by-stroke, with strokes deleted in the reverse chronological order in which they were provided in the recognition unit. Figures 17A-17H show exemplary user interfaces for providing character-by-character deletion within a multi-character handwritten input in response to a deletion input from a user.

[0270] As shown in Figure 17A, a user provides multiple handwritten strokes 1702 in the handwriting input area 804 of the handwriting input user interface 802. Based on these currently accumulated strokes 1702, the user device presents three recognition results (e.g., results 1704, 1706, and 1708) in the candidate display area 806. As shown in Figure 17B, the user provides multiple additional strokes 1710 in the handwriting input area 806. The user device recognizes three new output characters and replaces the three previous recognition results 1704, 1706, and 1708 with three new recognition results 1712, 1714, and 1716. In some embodiments, as shown in Figure 17B, even though the user device has identified two separate recognition units from the current handwritten input (e.g., stroke 1702 and stroke 1710), the cluster of strokes 1710 does not correspond well to any known characters in the handwriting recognition module's repertoire. As a result, the candidate character identified for the recognition unit containing the stroke 1710 (e.g., " JPEG2025124752000120.jpg1011", " JPEG2025124752000121.jpg109") all have recognition confidence below a predefined threshold. In some embodiments, the user device may select candidate characters for the first recognition unit (e.g., " JPEG2025124752000122.jpg108") but not any candidate characters for the second recognition unit, is presented in candidate display area 806. In some embodiments, the user device also displays a full recognition result (e.g., result 1714 or 1716) that includes candidate characters for both recognition units, regardless of whether the recognition confidence passes a predefined threshold. Providing a partial recognition result informs the user of any portions of the handwritten input that require correction. Furthermore, the user can also choose to first enter the correctly recognized portions of the handwritten input and then rewrite the portions that were not correctly recognized.

[0271] 17C shows that the user continues to provide an additional handwritten stroke 1718 to the left of stroke 1710. Based on the relative location and distance of stroke 1718, the user device determines that this newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes 1702. Based on this corrected recognition unit, the user device generates a new character (e.g., ") for the first recognition unit. The first recognition result 1720 is again a partial recognition result because none of the candidate characters identified for stroke 1710 meet the predefined confidence threshold.

[0272] FIG. 17D shows that the user has now entered multiple new strokes 1726 between stroke 1702 and stroke 1710. The user device assigns the newly entered strokes 1726 to the same recognition unit as stroke 1710. At this point, the user has entered two Chinese characters (e.g., " 17. All handwritten strokes for the image “JPEG2025124752000124.jpg1017” have been input, and the correct recognition result 1728 is displayed in the candidate display area 806.

[0273] 27E shows the user entering the initial portion of a delete input, for example, by performing a light touch 1730 on delete button 1732. If the user maintains contact with delete button 1732, the user can delete the current handwritten input character by character (or by recognition unit). This deletion is not performed for all handwritten input simultaneously.

[0274] In some embodiments, when a user's finger first touches the delete button 1732 on the touch-sensitive screen, the last recognition unit (e.g., the character " JPEG2025124752000125.jpg1312) is visually highlighted (e.g., by border 1734 or a light background) relative to other recognition units simultaneously displayed within handwriting input area 804.

[0275] In some embodiments, when the user device detects that the user maintains contact 1730 on delete button 1732 for more than a threshold duration, the user device removes the highlighted recognition unit (e.g., in box 1734) from handwriting input area 806, as shown in Figure 17F. Furthermore, the user device also corrects the recognition result shown in candidate display area 608 to delete any output characters generated based on the deleted recognition unit, as shown in Figure 17F.

[0276] FIG. 17F shows the last recognition unit (e.g., the character " If the user continues to maintain contact 1730 on the delete button 1732 after a recognition unit related to "JPEG2025124752000126.jpg1312" has been deleted, a recognition unit adjacent to the deleted recognition unit (e.g., a recognition unit related to the character " 17F , this remaining recognition unit is visually highlighted (e.g., in box 1736) and is ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. If the user discontinues contact with the delete button before the threshold duration is reached, the visual highlighting is removed from the last recognition unit and that recognition unit is not deleted. As will be recognized by those skilled in the art, the duration of contact is reset each time a recognition unit is deleted. Furthermore, in some embodiments, the strength of the contact (e.g., the pressure the user is applying to contact 1730 with the touch-sensitive screen) is optionally used to adjust the threshold duration for confirming the user's intent to delete the currently highlighted recognition unit. 17F and 17G show that the user discontinues contact 1730 with the delete button 1732 before the threshold duration is reached, and the text " 17G and 17H, a recognition unit for "JPEG2025124752000128.jpg99" is stored in handwriting input area 806. When the user selects the first recognition result (e.g., result 1738) for this recognition unit (e.g., as indicated by contact 1740), the text in first recognition result 1738 is entered into text entry area 808.

[0277] 18A and 18B are flow diagrams of an example process 1800 in which a user device provides character-by-character deletion within a multi-character handwritten input. In some embodiments, this deletion of the handwritten input is performed before the characters recognized from the handwritten input are identified and entered into the text entry area of ​​the user interface. In some embodiments, the deletion of characters within the handwritten input proceeds in reverse spatial order of the recognition units identified from the handwritten input, regardless of the temporal sequence in which the recognition units are formed. FIGS. 17A-17H illustrate an example process 1800, according to some embodiments.

[0278] As shown in FIG. 18A , in example process 1800, a user device receives handwritten input from a user (1802), the handwritten input including a plurality of handwritten strokes provided in a handwriting input area (e.g., area 804 in FIG. 17D ) of a handwriting input interface. The user device identifies (1804) a plurality of recognition units from the plurality of handwritten strokes, each recognition unit including a corresponding subset of the plurality of handwritten strokes. For example, as shown in FIG. 17D , a first recognition unit includes strokes 1702 and 1718, and a second recognition unit includes strokes 1710 and 1726. The user device generates (1806) a multi-character recognition result (e.g., result 1728 in FIG. 17D ) including respective characters recognized from the plurality of recognition units. In some embodiments, the user device displays the multi-character recognition result (e.g., result 1728 in FIG. 17D ) in a candidate display area of ​​the handwriting input interface. In some embodiments, while the multi-character recognition result is displayed in the candidate display area, the user device receives 1810 a delete input from the user (e.g., contact 1730 on delete button 1732), as shown in, for example, FIG. 17E. In some embodiments, in response to receiving this delete input, the user device deletes the last character (e.g., the spatial arrangement " The character " appears at the end of "JPEG2025124752000129.jpg1020". Remove "JPEG2025124752000130.jpg1312" (1812).

[0279] In some embodiments, the user device renders (1814) multiple handwritten strokes in real time within a handwriting input area of ​​a handwriting input interface as the multiple handwritten strokes are provided by a user, as shown, for example, in FIGS. 17A-17D. In some embodiments, in response to receiving a delete input, the user device removes (1816) from the handwriting input area (e.g., handwriting input area 804 in FIG. 17E) a corresponding subset of the multiple handwritten strokes that corresponds to the last recognition unit (e.g., the recognition unit including strokes 1726 and 1710) in a spatial array formed by the multiple recognition units within the handwriting input area. This last recognition unit is used to identify the last character (e.g., the character " )" in a multi-character recognition result (e.g., result 1728 in FIG. 17E). JPEG2025124752000131.jpg1312" is supported.

[0280] In some embodiments, this last recognition unit does not include the last handwritten stroke in time among the multiple handwritten strokes provided by the user 1818. For example, if the user provides stroke 1718 after providing strokes 1726 and 1710, the last recognition unit including strokes 1726 and 1710 is still deleted first.

[0281] In some embodiments, in response to receiving the initial portion of the delete input, the user device visually distinguishes 1820 the last recognition unit from other recognition units identified within the handwriting input area, for example, as shown in Figure 17E. In some embodiments, the initial portion of the delete input is an initial contact detected 1822 on a delete button within the handwriting input interface, and the delete input is detected if this initial contact is sustained beyond a predetermined threshold amount of time.

[0282] In some embodiments, the last recognition unit corresponds to a handwritten Chinese character. In some embodiments, the handwritten input is written in a cursive script. In some embodiments, the handwritten input corresponds to multiple Chinese characters written in a cursive script. In some embodiments, at least one of the handwritten strokes is separated into two adjacent recognition units of the multiple recognition units. For example, in some cases, a user may use a long stroke that flows into multiple characters, and in such cases, a segmentation module of the handwriting input module optionally separates the long stroke into several recognition units. When deletion of the handwritten input is performed character by character (or recognition unit by recognition unit), only a segment of the long stroke (e.g., a segment within a corresponding recognition unit) is deleted at a time.

[0283] In some embodiments, the delete input is a sustained contact on a delete button provided within the handwriting input interface (1824), and removing the corresponding subset of the plurality of handwritten strokes further includes removing the subset of handwritten strokes in the last recognition unit from the handwriting input area, stroke by stroke, in the reverse chronological order in which the subset of handwritten strokes was provided by the user.

[0284] In some embodiments, the user device generates 1826 partial recognition results including a subset of each character recognized from the multiple recognition units, each of which satisfies a predetermined confidence threshold, as shown, for example, in Figures 17B and 17C. In some embodiments, the user device displays 1828 the partial recognition results (e.g., result 1712 in Figure 17B and result 1720 in Figure 17C) simultaneously with the multi-character recognition results (e.g., results 1714 and 1722) in a candidate display area of ​​the handwriting input interface.

[0285] In some embodiments, the partial recognition result does not include at least the last character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the first character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the middle character in the multi-character recognition result.

[0286] In some embodiments, the smallest unit of deletion is the radical, and the handwriting is deleted one radical at a time whenever the radical happens to be the last recognition unit in the handwriting that still remains in the handwriting area.

[0287] As described herein, in some embodiments, the user device provides both a horizontal writing mode and a vertical writing mode. In some embodiments, the user device allows a user to input text in one or both of a left-to-right writing direction and a right-to-left direction in the horizontal writing mode. In some embodiments, the user device allows a user to input text in one or both of a top-to-bottom writing direction and a bottom-to-top direction in the vertical writing mode. In some embodiments, the user device provides various affordances (e.g., writing mode or writing direction buttons) on the user interface to invoke the respective writing mode and / or writing direction for the current handwriting input. In some embodiments, the text input direction in the text entry area is the same as the handwriting input direction by default. In some embodiments, the user device allows a user to manually set the input direction in the text entry area and the writing direction in the handwriting input area. In some embodiments, the text display direction in the candidate display area is the same as the handwriting input direction by default. In some embodiments, the user device allows a user to manually set the direction of text display in the text entry area, independent of the direction of handwriting input in the handwriting entry area. In some embodiments, the user device associates a writing mode and / or writing direction of the handwriting input interface with a corresponding device orientation, and a change in the device orientation triggers a change in writing mode and / or writing direction. In some embodiments, a change in writing direction automatically inputs the top-ranked recognition result into the text entry area.

[0288] 19A-19F illustrate an exemplary user interface of a user device that provides both horizontal and vertical input modes.

[0289] 19A shows a user device in horizontal input mode. In some embodiments, horizontal input mode is provided when the user device is in a landscape orientation, as shown in FIG. 19A. In some embodiments, horizontal input mode is optionally associated with and provided when the device is operated in a portrait orientation. In different applications, the association between device orientation and writing mode may be different.

[0290] In horizontal input mode, a user can provide handwritten characters in a horizontal writing direction (e.g., a default writing direction that runs from left to right or a default writing direction that runs from right to left), and the user device performs segmentation of the handwritten input into one or more recognition units along this horizontal writing direction.

[0291] In some embodiments, the user device allows only a single line of input within the handwriting input area. In some embodiments, the user device allows multiple lines of input (e.g., two lines of input) within the handwriting input area, as shown in FIG. 19A . In FIG. 19A , the user provides handwritten strokes in several lines within the handwriting input area 806. Based on the arrangement of the handwritten strokes provided by the user, as well as the relative locations of the handwritten strokes and the distances between them, the user device determines that the user is inputting two lines of characters. After dividing the handwritten input into two separate lines, the device determines the recognition units within each line.

[0292] As shown in FIG. 19A , the user device recognizes each character for each recognition unit identified in the current handwritten input 1902, generating several recognition results 1904 and 1906. As further shown in FIG. 19A , in some embodiments, if the output character (e.g., the letter “I”) for a particular set of recognition units (e.g., recognition units formed by initial strokes) is low, the user device optionally generates a partial recognition result (e.g., result 1906) that indicates only output characters with sufficient recognition confidence. In some embodiments, the user can understand from this partial recognition result 1906 that the first stroke can be corrected or individually deleted in order for the recognition model to produce a correct recognition result. In this particular example, editing the first recognition unit 1904 is not necessary because it indicates the desired recognition result for the first recognition unit.

[0293] In this example, as shown in FIGS. 19A and 19B, a user rotates a device to a portrait orientation (e.g., as shown in FIG. 19B). In response to this change in device orientation, the handwriting input interface changes from a horizontal input mode to a vertical input mode, as shown in FIG. 19B. In the vertical input mode, the layout of the handwriting input area 804, the candidate display area 806, and the text input area 808 may differ from that shown in the horizontal input mode. The specific layouts of the horizontal and vertical input modes can be varied to suit the requirements of various device shapes and applications. In some embodiments, the rotation of the device orientation and the change in input mode cause the user device to automatically enter the top-ranked result (e.g., result 1904) as text entry 1910 in the text input area 808. The orientation and position of cursor 1912 also reflect the change in input mode and writing direction.

[0294] In some embodiments, the change in input mode is optionally triggered by the user touching a special input mode selection affordance 1908. In some embodiments, this input mode selection affordance is a graphical user interface element that also indicates the current writing mode, the current writing direction, and / or the current paragraph direction. In some embodiments, the input mode selection affordance can cycle through all available input modes and writing directions provided by the handwriting input interface 802. As shown in FIG. 19A , affordance 1908 indicates that the current input mode is a horizontal input mode with a left-to-right writing direction and a top-to-bottom paragraph direction. In FIG. 19B , affordance 1908 indicates that the current input mode is a vertical input mode with a top-to-bottom writing direction and a right-to-left paragraph direction. Other combinations of writing direction and paragraph direction are possible according to various embodiments.

[0295] As shown in FIG. 19C, the user, in vertical input mode, writes multiple new strokes 1914 (e.g., two Chinese characters " The user inputs handwritten strokes related to "JPEG2025124752000132.jpg1019." The handwritten input is written in a vertical writing direction. The user device divides the vertical handwritten input into two recognition units and displays two recognition results 1916 and 1918, each containing two vertically laid out recognized characters.

[0296] 19C and 19D show that when a user selects a displayed recognition result (e.g., result 1916), the selected recognition result is entered vertically into text entry area 808.

[0297] 19E and 19F show the user entering additional lines of handwriting input 1920 in a vertical writing direction. These lines proceed from left to right, following the paragraph direction of traditional Chinese writing. In some embodiments, the candidate display area 806 also shows recognition results (e.g., results 1922 and 1924) in the same writing direction and paragraph direction as that for the handwriting input area. In some embodiments, other writing directions and paragraph directions may be provided by default according to the primary language associated with the user device or the language of a soft keyboard installed on the user device (e.g., Arabic, Chinese, Japanese, English, etc.).

[0298] 19E and 19F also show that when the user selects a recognition result (e.g., result 1922), the text of the selected recognition result is entered into text entry area 808. As shown in Figure 19F, the text currently entered in text entry area 808 therefore includes both text written in horizontal mode, which has a left-to-right writing direction, and text written in vertical mode, which has a top-to-bottom writing direction. The paragraph direction for horizontal text is top-to-bottom, while the paragraph direction for vertical text is right-to-left.

[0299] In some embodiments, the user device allows the user to establish a preferred writing direction and paragraph direction separately for each of the handwriting input area 804, the candidate display area 806, and the text input area 808. In some embodiments, the user device allows the user to establish a preferred writing direction and paragraph direction associated with the respective device orientation for each of the handwriting input area 804, the candidate display area 806, and the text input area 808.

[0300] Figures 20A-20C are a flow diagram of an example process 2000 for changing the text input direction and handwriting input direction of a user interface. Figures 19A-19F illustrate the process 2000, according to some embodiments.

[0301] In some embodiments, a user device determines a device orientation (2002). Device orientation and changes in device orientation can be detected by an accelerometer and / or other orientation sensing element within the user device. In some embodiments, the user device provides a handwriting input interface on the device in a horizontal input mode according to the device being in a first orientation (2004). Each line of handwriting input entered in the horizontal input mode is divided into one or more respective recognition units along a horizontal writing direction. In some embodiments, the device provides a handwriting input interface on the device in a vertical input mode according to the device being in a second orientation (2006). Each line of handwriting input entered in the vertical input mode is divided into one or more respective recognition units along a vertical writing direction.

[0302] In some embodiments, while operating in a horizontal input mode (2008), the device detects a change in device orientation from a first orientation to a second orientation (2010). In some embodiments, in response to the change in device orientation, the device switches from the horizontal input mode to a vertical input mode (2012). This is shown, for example, in FIGS. 19A and 19B. In some embodiments, while operating in a vertical input mode (2014), the user device detects a change in device orientation from a second orientation to a first orientation (2016). In some embodiments, in response to the change in device orientation, the user device switches from the vertical input mode to the horizontal input mode (2018). In some embodiments, the association between device orientation and input mode can be reversed from that described above.

[0303] In some embodiments, while operating in a horizontal input mode (2020), the user device receives a first multi-word handwritten input from a user (2022). In response to the first multi-word handwritten input, the user device presents recognition results for the first multi-word in a candidate display area of ​​the handwriting input interface according to a horizontal writing direction (2024). This is shown, for example, in FIG. 19A . In some embodiments, while operating in a vertical input mode (2026), the user device receives a second multi-word handwritten input from a user (2028). In response to the second multi-word handwritten input, the user device presents recognition results for the second multi-word in a candidate display area according to a vertical writing direction (2030). This is shown, for example, in FIGS. 19C and 19E .

[0304] In some embodiments, the user device receives (2032) a first user input selecting a first multi-word recognition result, for example, as shown in Figures 19A and 19B, where the selection is implicitly performed using an input to change the input direction (e.g., rotating the device or selecting affordance 1908). The user device receives (2034) a second user input selecting a second multi-word recognition result, for example, as shown in Figures 19C or 19E. The user device simultaneously displays (2036) corresponding text of the first multi-word recognition result and the second multi-word recognition result in a text input area of ​​the handwriting input interface, where the corresponding text of the first multi-word recognition result is displayed according to a horizontal writing direction and the corresponding text of the second multi-word recognition result is displayed according to a vertical writing direction. This is shown, for example, in text input area 808 in Figure 19F.

[0305] In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a horizontal writing direction and has a default top-to-bottom paragraph direction. In some embodiments, the horizontal writing direction is left-to-right. In some embodiments, the horizontal writing direction is right-to-left. In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area accepts multiple lines of handwriting input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, the vertical writing direction is top-to-bottom. In some embodiments, the first orientation is a default landscape orientation and the second orientation is a default portrait orientation. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface for manually switching between horizontal and vertical input modes regardless of the device orientation. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface for manually switching between two alternative writing directions. In some embodiments, the user device provides a corresponding affordance within the handwriting input interface for manually switching between two alternative paragraph directions. In some embodiments, this affordance is a toggle button that, when invoked one or more times in succession, cycles through each available combination of input direction and paragraph direction.

[0306] In some embodiments, the user device receives handwritten input from a user (2038). The handwritten input includes a plurality of handwritten strokes provided within a handwriting input area of ​​the handwriting input interface. In response to the handwritten input, the user device displays one or more recognition results within a candidate display area of ​​the handwriting input interface (2040). While the one or more recognition results are displayed within the candidate display area, the user device detects user input to switch from the current handwriting input mode to an alternative handwriting input mode (2042). In response to the user input (2044), the user device switches from the current handwriting input mode to the alternative handwriting input mode (2046). In some embodiments, the user device erases the handwritten input from the handwriting input area (2048). In some embodiments, the user device automatically enters (2050) the highest-ranked recognition result of the one or more recognition results displayed within the candidate display area into a text input area of ​​the handwriting input interface. 19A and 19B, where the current handwriting input mode is a horizontal input mode and the alternate handwriting input mode is a vertical input mode. In some embodiments, the current handwriting input mode is a vertical input mode and the alternate handwriting input mode is a horizontal input mode. In some embodiments, the current handwriting input mode and the alternate handwriting input mode are modes in which any two different handwriting or paragraph directions are provided. In some embodiments, the user input is a rotation of the device from the current orientation to a different orientation (2052). In some embodiments, the user input is the invocation of an affordance to manually switch the current handwriting input mode to the alternate handwriting input mode.

[0307] As described herein, the handwriting input module allows a user to input handwritten strokes and / or characters in any temporal order. Therefore, deleting individual handwritten characters within a multi-character handwritten input and rewriting the same or a different handwritten character in the same place as the deleted character is advantageous because it helps a user correct a long handwritten input without having to delete the entire handwritten input.

[0308] 20A-20H show exemplary user interfaces for visually highlighting and / or deleting recognition units identified within multiple handwritten strokes currently stored in a handwritten input area. Enabling a user to individually select, view, and delete any one of multiple recognition units identified within multiple inputs is particularly useful when multi-character or even multi-line handwritten input is permitted by the user device. Enabling a user to delete a specific recognition unit at the beginning or middle of a handwritten input allows a user to make corrections to a long input without requiring the user to delete all recognition units located after the undesired recognition unit.

[0309] 21A-21C, a user provides multiple handwritten strokes (e.g., strokes 2102, 2104, and 2106) in handwriting input area 804 of handwriting input user interface 802. As the user continues to provide additional strokes in handwriting input area 804, the user device updates the recognition units identified from the handwritten input currently stored in the handwriting input area and corrects the recognition results according to the output characters recognized from the updated recognition units. As shown in FIG. 20C, the user device identifies two recognition units from the current handwritten input and presents three recognition results (e.g., 2108, 2110, and 2112), each containing two Chinese characters.

[0310] In this example, after the user writes two handwritten characters, the user realizes that the first recognition unit was written incorrectly and, as a result, the user device has not identified and presented the desired recognition result in the candidate display area.

[0311] In some embodiments, when a user provides a tap gesture on the touch-sensitive display (e.g., contact followed by an immediate lift-off in the same location), the user device interprets the tap gesture as an input to cause a visual highlighting of the individual recognition unit currently identified within the writing area. In some embodiments, another predefined gesture (e.g., a multi-finger wipe gesture over the writing area) is used to cause the user device to highlight the individual recognition unit within the writing area 804. Tap gestures may be preferred because they are relatively easy to distinguish from handwritten strokes (which typically involve a longer duration of sustained contact and have contact movement within the writing area 804). Multi-tap gestures may be preferred because they are relatively easy to distinguish from handwritten strokes (which typically involve a single contact within the writing area 804). In some embodiments, the user device provides an affordance 2112 within the user interface that can be invoked by the user (e.g., via contact 2114) to cause individual recognition units to be visually highlighted (e.g., as shown by boxes 2108 and 2110). In some embodiments, this affordance is preferred when there is sufficient screen space to accommodate such an affordance. In some embodiments, this affordance can be invoked multiple times in succession by the user, which causes the user device to visually highlight identified recognition units according to different division chains within the division bundle, and to turn off the highlighting when all division chains have been shown.

[0312] As shown in FIG. 21D , when the user provides the necessary gesture to highlight individual recognition units within the handwriting input area 804, the user device further displays a respective delete affordance (e.g., small delete buttons 2116 and 2118) above each highlighted recognition unit. FIGS. 21E and 21F show that when the user touches (e.g., via contact 2120) the respective recognition unit's delete affordance (e.g., delete button 2116 for the first recognition unit within box 2118), the respective recognition unit (e.g., within box 2118) is removed from the handwriting input area 804. In this particular example, the deleted recognition unit is neither the last recognition unit entered in time nor the last recognition unit spatially along the writing direction. In other words, the user can delete any recognition unit, regardless of when and where that recognition unit was provided within the handwriting input area. 21F shows that in response to the deletion of this first recognition unit in the handwriting input area, the user device also updates the recognition result displayed in candidate display area 806. As shown in FIG. 21F, the user device also deletes the candidate character corresponding to the deleted recognition unit from the recognition result. As a result, a new recognition result 2120 is displayed in candidate display area 806.

[0313] 21G and 21H, after a first recognition unit is removed from handwriting input interface 804, the user provides multiple new handwritten strokes 2122 in the area previously occupied by the removed recognition unit. The user device subdivides the currently stored handwritten input in handwriting input area 804. Based on the recognition units identified from the handwritten input, the user device regenerates recognition results (e.g., results 2124 and 2126) in candidate display area 806. Figures 21G and 21H show that the user selects one of the recognition results (e.g., result 2124) (e.g., via contact 2128) and the text of the selected recognition result is entered into text input area 808.

[0314] 22A and 22B are flow diagrams of an example process 2200 in which individual recognition units identified within a current handwritten input are visually presented and can be individually deleted, regardless of the temporal order in which they were formed.

[0315] In example process 2200, a user device receives handwritten input from a user (2202). The handwritten input includes a plurality of handwritten strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwritten strokes (2204) in a handwriting input area (e.g., handwriting input area 804) of a handwriting input interface. In some embodiments, the user device divides the plurality of handwritten strokes (2206) into two or more recognition units, each recognition unit including a corresponding subset of the plurality of handwritten strokes.

[0316] In some embodiments, the user device receives an edit request from a user (2208). In some embodiments, the edit request is a contact (2210) detected over a predefined affordance (e.g., affordance 2112 in FIG. 21D ) provided within the handwriting input interface. In some embodiments, the edit request is a tap gesture (2212) detected over a predefined area within the handwriting input interface. In some embodiments, the predefined area is within the handwriting input area of ​​the handwriting input interface. In some embodiments, the predefined area is outside the handwriting input area of ​​the handwriting input interface. In some embodiments, another predefined gesture outside the handwriting input area (e.g., a cross gesture, a horizontal swipe gesture, a vertical swipe gesture, a diagonal swipe gesture) can be used as the edit request. Gestures outside the handwriting input area can be easily distinguished from handwritten strokes because they are provided outside the handwriting input area.

[0317] In some embodiments, in response to the edit request, the user device visually identifies 2214 two or more recognition units within the handwriting input area, for example, using boxes 2108 and 2110 in Figure 21D. In some embodiments, visually identifying the two or more recognition units further includes highlighting 2216 respective boundaries between the two or more recognition units within the handwriting input area. Various embodiments may use different methods of visually identifying the recognition units identified within the current handwriting input.

[0318] In some embodiments, the user device provides means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units is a respective delete button displayed near each recognition unit, as shown, for example, by delete buttons 2116 and 2118 in FIG. 21D . In some embodiments, the means for individually deleting each of the two or more recognition units is a means for detecting a default delete gesture input over each recognition unit. In some embodiments, the user device does not visually display a separate delete affordance over the highlighted recognition unit. Instead, in some embodiments, the user can use a delete gesture to delete each recognition unit below it. In some embodiments, when the user device displays the recognition units in a visually highlighted manner, the user device does not accept further handwritten strokes within the handwriting input area. Instead, a default gesture or any gesture detected over a visually highlighted recognition unit causes the user device to remove that recognition unit from the handwriting input area and correct the recognition results displayed in the candidate display area accordingly. In some embodiments, a tap gesture causes the user device to visually highlight individual recognition units identified in the handwriting recognition area, and the user can then use a delete button to delete the individual recognition units one by one in the reverse direction of writing.

[0319] In some embodiments, the user device receives a deletion input from the user through the provided means (2224) to individually delete a first recognition unit of two or more recognition units from the handwriting input area, for example, as shown in FIG. 21E. In response to the deletion input, the user device removes a corresponding subset of handwritten strokes within the first recognition unit from the handwriting input area (2226), for example, as shown in FIG. 21F. In some embodiments, the first recognition unit is a spatially first recognition unit within the two or more recognition units. In some embodiments, the first recognition unit is a spatially middle recognition unit within the two or more recognition units, for example, as shown in FIGS. 21E and 21F. In some embodiments, the first recognition unit is a spatially last recognition unit within the two or more recognition units.

[0320] In some embodiments, the user device generates (2228) a segment lattice from the plurality of handwritten strokes, the segment lattice including multiple alternative segment chains, each representing a set of corresponding recognition units identified from the plurality of handwritten strokes. For example, FIG. 21G shows recognition results 2024 and 2026, where recognition result 2024 is generated from one segment chain having two recognition units and recognition result 2026 is generated from another segment chain having three recognition units. In some embodiments, the user device receives (2230) two or more successive edit requests from the user. For example, the two or more successive edit requests can be several successive taps on affordance 2112 in FIG. 21G. In some embodiments, in response to each of the two or more successive edit requests, the user device visually identifies (2232) within the handwriting input area a set of corresponding recognition units from a different one of the multiple alternative segment chains. For example, in response to a first tap gesture, two taps (e.g., the characters " JPEG2025124752000133.jpg1010" and " JPEG2025124752000134.jpg99) recognition units are highlighted, and in response to the second tap gesture, three (e.g., the characters " JPEG2025124752000135.jpg108", " JPEG2025124752000136.jpg98" and " JPEG2025124752000137.jpg99) are highlighted. In some embodiments, in response to a third tap gesture, the visual highlighting is optionally removed from all recognition units, and the writing area returns to a normal state, ready to accept further strokes. In some embodiments, the user device provides means for individually deleting each of the set of corresponding recognition units currently represented in the writing area (2234). In some embodiments, this means is an individual delete button for each highlighted recognition unit. In some embodiments, this means is means for detecting a predefined delete gesture over each highlighted recognition unit and invoking a function to delete the highlighted recognition unit below the predefined delete gesture.

[0321] As described herein, in some embodiments, a user device provides a continuous input mode within a writing area. Because the area of ​​a writing area is limited on a portable user device, it may be desirable to provide a method for caching handwritten input provided by a user to allow the user to reuse screen space without committing previously provided handwritten input. In some embodiments, a user device provides a scrolling handwritten input area, in which the input area gradually shifts a specific amount (e.g., one recognition unit at a time) when the user is sufficiently close to the end of the handwritten input area. In some embodiments, because shifting existing recognition units within the handwritten input area can disrupt the user's writing process and possibly prevent correct segmentation of the recognition units, it may be advantageous to reclaim previously used areas of the input area without dynamically shifting the recognition units. In some embodiments, when a user reuses an area occupied by handwritten input that has not yet been entered into the text input area, the top-ranked recognition result for that handwritten input area is automatically entered into the text input area, allowing the user to continue providing new handwritten input without explicitly selecting the top-ranked recognition result.

[0322] Some conventional systems allow a user to write over existing handwritten input that is still shown in the handwriting input area. Such systems use temporal information to determine whether a new stroke is part of a previous recognition unit or a new recognition unit. Systems that rely on such temporal information impose strict requirements on the speed and cadence with which a user provides handwritten input, requirements that are difficult for many users to meet. Furthermore, the visual rendering of the handwritten input can be cluttered and difficult for the user to decipher. Therefore, the writing process can be frustrating and confusing for the user, resulting in a poor user experience.

[0323] As described herein, a fading process is used to indicate when a user can reuse an area occupied by a previously written recognition unit and continue writing in the handwriting input area. In some embodiments, this fading process gradually reduces the visibility of each recognition unit provided in the handwriting input area over a threshold amount of time so that if a new stroke is written over it, the existing text does not visually compete with the new stroke. In some embodiments, writing over a faded recognition unit causes the top-ranked recognition result for that recognition unit to be automatically entered into the text input area without the user having to interrupt writing and explicitly provide selection input for the top-ranked recognition result. This implicit and automatic confirmation of the top-ranked recognition result improves the efficiency and speed of input in handwriting input interfaces and reduces the cognitive burden imposed on the user to maintain a train of thought for the current text composition. In some embodiments, writing over a faded recognition unit does not cause automatic selection of the top-ranked search result. Instead, the faded recognition units are cached in the handwriting stack and combined with the new handwriting as the current handwriting, and the user can view the recognition results generated based on all of the recognition units accumulated in the handwriting stack before making a selection.

[0324] 23A-23J show an exemplary user interface and process in which recognition units provided in different regions of a handwriting input area gradually fade out from their corresponding regions, e.g., after a predetermined amount of time, and after this fade-out occurs in a particular region, the user is able to provide new handwritten strokes in that region.

[0325] 23A , a user provides multiple handwritten strokes 2302 (e.g., three handwritten strokes for a capital letter “I”) in the handwriting input area 804. The handwritten strokes 2302 are identified as a recognition unit by the user device. In some embodiments, the handwritten input currently shown in the handwriting input area 804 is cached in a first tier in the handwriting input stack of the user device. Several recognition results generated based on the identified recognition units are provided in the candidate display area 804.

[0326] FIG. 23B shows that as the user continues to write one or more strokes 2302 to the right of stroke 2304, handwritten stroke 2302 in the first recognition unit begins to gradually fade out in the handwriting input area 804. In some embodiments, an animation is displayed that mimics the gradual fading or disappearance of the visual rendering of this first recognition unit. For example, this animation can create the visual effect of ink evaporating from a whiteboard. In some embodiments, the fading of the recognition unit is not uniform throughout the recognition unit. In some embodiments, the fading of the recognition unit increases over time until the recognition unit is completely invisible in the handwriting area. However, even when the recognition unit is no longer visible in the handwriting input area 804, in some embodiments, the invisible recognition unit remains at the top of the handwriting stack, and recognition results generated from the recognition unit continue to be displayed in the candidate display area. In some embodiments, the faded recognition unit is not completely removed from view until new handwritten input is written on top of it.

[0327] In some embodiments, the user device allows new handwriting input to be provided over the area formerly occupied by the faded recognition unit as soon as the fading animation begins, and in some embodiments, the user device allows new handwriting input to be provided over the area formerly occupied by the faded recognition unit only after the fading has progressed to a certain stage (e.g., to the faintest level or until recognition is completely invisible in the area).

[0328] 23C shows that the first recognition unit (i.e., stroke 2302) has completed its fading process (e.g., the ink color has stabilized to a very faint level or has become invisible). The user device is identifying additional recognition units (e.g., recognition units for the handwritten characters "a" and "m") from additional handwritten strokes provided by the user and updating the recognition results presented in candidate display area 804.

[0329] 22D-22F show that over time, the user provides multiple additional handwritten strokes (e.g., 2304 and 2306) within the handwriting input area 804. At the same time, previously identified recognition units gradually fade away from the handwriting input area 804. In some embodiments, each recognition unit takes a predetermined amount of time to begin its own fading process after it is identified. In some embodiments, the fading process for each recognition unit does not begin until the user begins inputting a second recognition unit downstream of that recognition unit. As shown in FIGS. 23B-23F, when handwritten input is provided cursively, a single stroke (e.g., stroke 2304 or stroke 2306) may continue through multiple recognition units within the handwriting input area (e.g., recognition units for each handwritten character in the word "am" or "back").

[0330] FIG. 22G illustrates that even after a recognition unit has begun its fading process, the user can restore the recognition unit to a non-faded state by a predefined playback input, for example, a tap gesture on the delete button 2310 (e.g., as indicated by contact 2308 followed by an immediate lift-off). Once a recognition unit is played back, its appearance returns to its normal visibility level. In some embodiments, playback of faded recognition units is performed character-by-character in the reverse direction of writing within the handwriting input area 804. In some embodiments, playback of faded recognition units is performed word-by-word within the handwriting input area 804. As shown in FIG. 23G, the recognition unit corresponding to the word "back" has been played back from a faded state to a fully non-faded state. In some embodiments, the clock for initiating the fading process is reset for each recognition unit as it is played back to a non-faded state.

[0331] FIG. 22H shows that sustained contact on the delete button deletes the last recognition unit in the default writing direction (e.g., the recognition unit for the letter "k" in the word "back") from the handwriting input area 804. As this delete input is continuously maintained, more recognition units (e.g., the recognition units for the letters "c," "a," and "b" in the word "back") are deleted one by one in the reverse writing direction. In some embodiments, the deletion of recognition units is word-by-word, and all characters of the handwritten word "back" that are deleted from the handwriting input area 804 are removed simultaneously. FIG. 22H also shows that when contact 2308 is maintained on the delete button 2310 after the deletion of the recognition unit for the letter "b" in the handwritten word "back," the previously faded recognition unit "m" is similarly regenerated.

[0332] 23I shows that if the deletion input is stopped before the deletion of the retrieved recognition unit "m" in the handwritten word "am" is performed, the retrieved recognition unit will again gradually fade. In some embodiments, the state of each recognition unit (e.g., a state selected from a set of one or more fade and non-fade states) is maintained and updated in a handwriting input stack.

[0333] FIG. 23J illustrates that, when a user provides one or more strokes 2312 over an area in the handwriting input area that was previously occupied by a faded recognition unit (e.g., the recognition unit for the letter "I"), in some embodiments, the text of the top-ranked recognition result (e.g., result 2314) for the handwriting input performed before stroke 2312 is automatically entered into text input area 808, as shown in FIGS. 23I-J. As shown in FIG. 23J, the text "I am" is no longer indicated as tentative and instead is confirmed in text input area 808. In some embodiments, when text input is performed for a completely faded or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly entered stroke (e.g., stroke 2312) becomes the current input in the handwriting input stack.

[0334] 23J, the text "I am" is no longer shown as tentative and is instead committed in text entry area 808. In some embodiments, for a fully faded or partially faded handwriting input, once text entry is performed, the handwriting input is removed from the handwriting input stack. The newly entered stroke (e.g., stroke 2312) becomes the current entry in the handwriting input stack.

[0335] In some embodiments, when stroke 2312 is provided in the handwriting input area over an area previously occupied by a faded recognition unit (e.g., the recognition unit for the letter "I"), the text of the top-ranked recognition result (e.g., result 2314) for the handwriting input performed before stroke 2312 is not automatically entered into text input area 808. Instead, the current handwriting input (both faded and non-faded) in handwriting input area 804 is cleared and cached in a handwriting stack. New stroke 2312 is added to the cached handwriting input in the handwriting stack. The user device determines a recognition result based on the entire handwriting input currently stored in the handwriting stack. The recognition results are displayed in the candidate display area. In other words, even if only a portion of the currently stored handwriting input is shown in handwriting input area 804, the recognition result is generated based on the entire handwriting input (both the visible portion and the portion that is no longer visible) cached in the handwriting stack.

[0336] Figure 23K shows that the user has entered an additional stroke 2316 into the handwriting input area 804, which fades over time. Figure 23L shows that a new stroke 2318 written over the faded strokes 2312 and 2316 has entered the text of the top recognition result 2320 for the faded strokes 2312 and 2316 into the text input area 808.

[0337] In some embodiments, the user optionally provides handwriting input in multiple lines, and in some embodiments, when multiple lines of input are valid, the same fading process can be used to clear the handwriting area for new handwriting input.

[0338] 24A and 24B are flow diagrams of an example process 2400 for providing a fading process in a handwriting input area of ​​a handwriting input interface. Figures 23A-23K illustrate the process 2400, according to some embodiments.

[0339] In some embodiments, the device receives 2402 a first handwritten input from a user. The first handwritten input includes a plurality of handwritten strokes, the plurality of handwritten strokes forming a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of ​​a handwriting input interface. In some embodiments, the user device renders 2404 each of the plurality of handwritten strokes within the handwriting input area as the handwritten strokes are provided by the user.

[0340] In some embodiments, the user device initiates a respective fading process for each of the plurality of recognition units after the recognition unit is fully rendered (2406). In some embodiments, during the respective fading process, the rendering of the recognition unit in the first handwritten input fades away. This, in some embodiments, is illustrated in Figures 23A-23F.

[0341] In some embodiments, the user device receives (2408) a second handwritten input from a user over an area of ​​the handwriting input zone that was occupied by a faded recognition unit of the plurality of recognition units, as shown, for example, in Figures 23I, 23J, and 23K-L. In some embodiments, in response to receiving (2410) this second handwritten input, the user device renders (2412) the second handwritten input within the handwriting input zone and erases (2414) all faded recognition units from the handwriting input zone. In some embodiments, all recognition units that were entered within the handwriting input zone before the second handwritten input are erased from the handwriting input zone, regardless of whether the recognition unit has begun its fading process. This is shown, for example, in Figures 23I, 23J, and 23K-L.

[0342] In some embodiments, the user device generates (2416) one or more recognition results for the first handwritten input. In some embodiments, the user device displays (2418) the one or more recognition results in a candidate display area of ​​the handwriting input interface. In some embodiments, in response to receiving the second handwritten input, the user device automatically enters (2420) the top-ranked recognition result displayed in the candidate display area into the text input area of ​​the handwriting input interface without user selection. This is shown, for example, in Figures 23I, 23J, and 23K-L.

[0343] In some embodiments, the user device stores (2422) an input stack including the first handwritten input and the second handwritten input. In some embodiments, the user device generates (2424) one or more multi-character recognition results, each including a corresponding spatial character arrangement recognized from a concatenation of the first handwritten input and the second handwritten input. In some embodiments, the user device displays (2426) the one or more multi-character recognition results in a candidate display area of ​​the handwriting input interface while the rendering of the second handwritten input replaces the rendering of the first handwritten input in the handwriting input area.

[0344] In some embodiments, the respective fading process for each recognition unit begins when a predetermined period of time has elapsed since that recognition unit was completed by the user.

[0345] In some embodiments, the fading process for each recognition unit begins after that recognition unit when the user begins inputting strokes for the next recognition unit.

[0346] In some embodiments, the end state of the respective fading process for each recognition unit is a state of predetermined minimum visibility for that recognition unit.

[0347] In some embodiments, the end state of the respective fading process for each recognition unit is a state of zero visibility for that recognition unit.

[0348] In some embodiments, after the last recognition unit in the first handwriting input fades, the user device receives (2428) a default playback input from the user. In response to receiving this default playback input, the user device restores (2430) the last recognition unit from the faded state to the non-faded state. This is shown, for example, in Figures 23F-23H. In some embodiments, this default playback input is an initial contact detected on a delete button provided in the handwriting input interface. In some embodiments, a sustained contact detected on the delete button deletes the last recognition unit from the handwriting input area and restores the penultimate recognition unit from the faded state to the non-faded state. This is shown, for example, in Figures 23G and 23H.

[0349] As described herein, a multi-script handwriting recognition model performs stroke-order- and stroke-direction-independent recognition of handwritten characters. In some embodiments, the recognition model is trained solely on spatially derived features contained in flat images of writing samples that correspond to various characters in the vocabulary of the handwriting recognition model. Because the images of writing samples do not contain any temporal information associated with the individual strokes contained within those images, the resulting recognition model is stroke-order- and stroke-direction-independent.

[0350] As described above, stroke-order and stroke-direction independent handwriting recognition offers many advantages over conventional recognition systems that rely on information related to the temporal production of characters (e.g., the temporal arrangement of strokes within those characters). However, in real-time handwriting recognition scenarios, temporal information associated with individual strokes is available, and it may be beneficial to utilize this information to improve the recognition accuracy of a handwriting recognition system. Below, techniques are described that integrate temporally derived stroke distribution information into the extraction of spatial features for a handwriting recognition model, but the use of this temporally derived stroke distribution information does not compromise the stroke-order and / or stroke-direction independence of the handwriting recognition system. Based on the stroke distribution information associated with various characters, disambiguation is possible between similar-looking characters that are produced with distinct sets of strokes.

[0351] In some embodiments, when handwriting input is converted into an input image (e.g., an input bitmap image) for a handwriting recognition model (e.g., a CNN), the temporal information associated with individual strokes is lost. For example, the Chinese character " For example, for "JPEG2025124752000138.jpg99," eight strokes (e.g., labeled #1 through #8 in FIG. 27) can be used to write this character. The sequence and direction of the strokes for this character provide several unique features associated with the character. A naive way to capture stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system would be to explicitly enumerate all possible permutations of stroke order and stroke direction in the training samples. However, even for characters of only moderate complexity, this permutation would amount to over one billion possibilities, making it impractical, if not impossible, to implement in practice. As described herein, a stroke distribution profile is generated for each writing sample that abstracts the time-series aspects of stroke generation (i.e., temporal information). The stroke distribution profile of these writing samples is trained to extract a set of temporally derived features, which are then combined with spatially derived features (e.g., from an input bitmap image) to improve recognition accuracy without affecting the stroke order and stroke direction independence of the handwriting recognition system.

[0352] As described herein, temporal information associated with a character is extracted by calculating various pixel distributions that characterize each handwritten stroke. Every handwritten stroke of a character produces a deterministic pattern (or profile) when projected in a given direction. While this pattern by itself may not be sufficient to unambiguously recognize the stroke, when combined with other similar patterns, it may be suitable for capturing specific characteristics unique to this particular stroke. This type of stroke representation can also be integrated with spatial feature extraction (e.g., input image-based feature extraction in CNNs) to provide orthogonal information that can be useful for disambiguating similar-looking characters within the repertoire of handwriting recognition models.

[0353] 25A and 25B are flow diagrams of an example process 2500 for integrating temporally and spatially derived features of a handwriting sample during training of a handwriting recognition model, where the resulting recognition model remains stroke order and stroke direction independent. In some embodiments, the example process 2500 is executed on a server device that provides the trained recognition model to a user device (e.g., portable device 100). In some embodiments, the server device includes one or more processors and memory, and the memory includes instructions that, when executed by the one or more processors, perform the process 2500.

[0354] In example process 2500, the device separately trains (2502) a set of spatially derived features and a set of temporally derived features of a handwriting recognition model, where the set of spatially derived features is trained on a corpus of training images, each of which is an image of a handwriting sample for a respective character in the output character set, and the set of temporally derived features is trained on a corpus of stroke distribution profiles, each of which numerically characterizes the spatial distribution of multiple strokes within the handwriting sample for a respective character in the output character set.

[0355] In some embodiments, separately training the set of spatially derived features further includes training 2504 a convolutional neural network having an input layer, an output layer, and multiple convolutional layers, including a first convolutional layer, a last convolutional layer, zero or more intermediate convolutional layers between the first and last convolutional layers, and a hidden layer between the last convolutional layer and the output layer. An exemplary convolutional network 2602 is shown in FIG. 26. This exemplary convolutional network 2602 can be implemented in substantially the same manner as the convolutional network 602 shown in FIG. 6. The convolutional network 2602 includes an input layer 2606, an output layer 2608, and multiple convolutional layers, including an initial convolutional layer 2610a, zero or more intermediate convolutional layers, a final convolutional layer 2610n, and a hidden layer 2614 between the final convolutional layer and the output layer 2608. The convolutional network 2602 also includes a kernel layer 2616 and a subsampling layer 2612, according to the arrangement shown in FIG. 6. Training of the convolutional network is based on images 2614 of writing samples in a training corpus 2604. Spatially derived features are obtained, and respective weights associated with the various features are determined by minimizing recognition errors for training samples in the training corpus. After training, these same features and weights are used to recognize new handwriting samples not present in the training corpus.

[0356] In some embodiments, separately training the set of temporally derived features further includes providing the plurality of stroke distribution profiles to a statistical model to determine a plurality of temporally derived parameters and respective weights for the plurality of temporally derived parameters for classifying each character in the output character set (2506). In some embodiments, as shown in FIG. 26, a stroke distribution profile 2620 is derived from each writing sample in a training corpus 2622. The training corpus 2622 optionally includes the same writing samples as the corpus 2604, but also includes temporal information associated with stroke production within each writing sample. The stroke distribution profile 2622 is provided to a statistical modeling process 2624, during which temporally derived features are extracted and respective weights for the various features are determined by minimizing recognition or classification errors based on a statistical modeling method (e.g., CNN, K-nearest neighbors, etc.). 26, the set of temporally derived features and their respective weights are converted into a set of feature vectors (e.g., feature vector 2626 or feature vector 2628) and introduced into respective layers within convolutional neural network 2602. The resulting network therefore includes spatially derived parameters and temporally derived parameters that are orthogonal to each other and jointly contribute to character recognition.

[0357] In some embodiments, the device combines (2508) the set of spatially derived features and the set of temporally derived features in a handwriting recognition model. In some embodiments, combining (2510) the set of spatially derived features and the set of temporally derived features in a handwriting recognition model includes introducing (2510) a plurality of spatially derived parameters and a plurality of temporally derived parameters into one of the convolutional layers or a hidden layer of a convolutional neural network. In some embodiments, the plurality of temporally derived parameters and their respective weights are introduced into a final convolutional layer of the convolutional neural network for handwriting recognition (e.g., final convolutional layer 2610n in FIG. 26). In some embodiments, the plurality of temporally derived parameters and their respective weights are introduced into a hidden layer of the convolutional neural network for handwriting recognition (e.g., hidden layer 2614 in FIG. 26).

[0358] In some embodiments, the device uses this handwriting recognition model to provide real-time handwriting recognition of the user's handwritten input (2512).

[0359] In some embodiments, the device generates 2514 a corpus of stroke distribution profiles from a plurality of writing samples, each of which corresponds to a character in an output character set 2516, and separately stores spatial information for each constituent stroke of the handwriting sample when that stroke was written. In some embodiments, to generate the corpus of stroke distribution profiles, the device performs 2518 the following steps:

[0360] For each of a plurality of handwriting samples (2520), the device identifies (2522) constituent strokes within the handwriting sample, and for each identified stroke of the handwriting sample, the device calculates (2524) a respective occupancy rate along each of a plurality of predetermined directions, where the occupancy rate is the ratio of the projected span of the stroke direction to the maximum projected span of the writing sample. For each identified stroke of the handwriting sample, the device also calculates (2526) a respective saturation rate for the stroke based on the ratio of the respective number of pixels within the stroke to the total number of pixels within the writing sample. The user device then generates (2528) a feature vector for the handwriting sample as a stroke distribution profile for the writing sample, the feature vector including the respective occupancy rate and the respective saturation rate of at least N strokes within the handwriting sample, where N is a predetermined natural number. In some embodiments, N is less than the maximum number of strokes observed in any single writing sample within the plurality of writing samples.

[0361] In some embodiments, for each of the plurality of handwriting samples, the device sorts the occupancy of each of the identified strokes in each of the predetermined directions in descending order, and includes only the N top-ranked occupancy and saturation writing samples in the feature vector for that writing sample.

[0362] In some embodiments, the plurality of predetermined orientations include a horizontal orientation, a vertical orientation, a positive 45 degree orientation, and a negative 45 degree orientation of the writing sample.

[0363] In some embodiments, the handwriting recognition model is used to provide real-time handwriting recognition of a user's handwritten input by a device that receives the user's handwritten input and, in response to receiving the user's handwritten input, provides handwriting recognition output to the user substantially simultaneously with receiving the handwritten input.

[0364] The character " shown in Figure 27 For purposes of illustration, example embodiments are described herein using "JPEG2025124752000139.jpg99." In some embodiments, each input image of a handwritten character is optionally normalized to a square. The span of each individual handwritten stroke (e.g., strokes #1, #2, ..., and #3) is measured when projected onto this square horizontally, vertically, at a diagonal angle of +45 degrees, and at a diagonal angle of -45 degrees. The span of each stroke S is recorded as x-span(i), y-span(i), c-span(i), and d-span(i), respectively, for the four projection directions. Furthermore, the maximum span observed across the entire image is also recorded. The maximum span of the character is recorded as x-span, y-span, c-span, and d-span, respectively, for the four projection directions. For purposes of illustration, four projection directions are optionally considered herein, although in principle, any arbitrary set of projections can be used in various embodiments. The maximum span (e.g., shown as x-span, y-span, c-span, and d-span) and the four projection directions are The spans (e.g., shown as x span (4), y span (4), c span (4), and d span (4)) of one of the strokes (e.g., stroke #4) in "JPEG2025124752000140.jpg99" are shown in FIG. 27.

[0365] In some embodiments, once the above spans are measured for all strokes 1 through 5 (5 being the number of distinct handwritten strokes associated with this input image), the respective occupancies along each projection direction are calculated. For example, for stroke S i The corresponding occupancy rate R along the x direction with respect to x (i) is R x (i) = x span (i) / x span. Similarly, the corresponding occupancies along other projection directions can be calculated, R y (i) = y span (i) / y span, R c (i) = c span (i) / c span, R d(i) = d span (i) / d span.

[0366] In some embodiments, the occupancy of all strokes in each direction is sorted separately in descending order, and thus a respective ranking of all strokes in the input image is obtained for each projection direction in terms of their occupancy in that direction. The ranking of strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is independent of the order and direction in which the strokes were produced in the writing sample. Therefore, this occupancy-based ranking is stroke-order and stroke-direction independent, temporally derived information.

[0367] In some embodiments, each stroke is given a relative weight that indicates its importance to the character as a whole. In some embodiments, this weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is called the saturation ratio associated with each stroke.

[0368] In some embodiments, a feature vector can be created for each stroke based on the occupancy and saturation of each stroke. For each character, a set of feature vectors containing 5S number of features is created. This set of features is called the stroke distribution profile of that character.

[0369] In some embodiments, only a predetermined number of top-ranked strokes are used in constructing a stroke distribution profile for each character. In some embodiments, this predetermined number of strokes is 10. Based on these top 10 strokes, 50 stroke-derived features are generated for each character. In some embodiments, these features are introduced in the final convolutional layer of the convolutional neural network or in subsequent hidden layers.

[0370] In some embodiments, during real-time recognition, an input image of a recognition unit is provided to a handwriting recognition model that has been trained with both spatially and temporally derived features. The input image is processed through each layer of the handwriting recognition model shown in FIG. 26. When the processing of this input image reaches a layer where a stroke distribution profile input is required (e.g., the final convolutional layer or a hidden layer), the stroke distribution profile of that recognition unit is introduced into that layer. This processing of the input image and stroke distribution profile continues until an output classification (e.g., one or more candidate characters) is provided at the output layer 2608. In some embodiments, stroke distribution profiles of all recognition units are calculated and provided as input to the handwriting recognition model along with the input images of those recognition units. In some embodiments, the input image of the recognition unit is first passed through the handwriting recognition model (without the benefit of temporally trained features). If two or more similar-looking candidate characters are identified with similar recognition confidence values, then the stroke distribution profile of that recognition unit is introduced into the handwriting recognition model at a layer (e.g., the final convolutional layer or a hidden layer) that is trained with temporally derived features. When the input image and stroke distribution profile of that recognition unit pass through the final layer of the handwriting recognition model, the two or more similar-looking candidate characters can be better distinguished by the differences in their stroke distribution profiles. Therefore, using temporally derived information about how recognition units are formed by individual handwritten strokes improves recognition accuracy without compromising the stroke order and stroke direction independence of the handwriting recognition system.

[0371] The foregoing description has been set forth with reference to specific embodiments for purposes of explanation. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, and to thereby enable others skilled in the art to best utilize the invention and various embodiments, with various modifications as suited to the particular uses contemplated.

Claims

1. 1. A method comprising: In an electronic device having a display and one or more input devices, detecting a plurality of handwritten strokes via the one or more input devices; displaying, via the display, a first string of text based on the plurality of handwritten strokes, the first string including first text at a first location and second text at a second location, the second location being different from the first location; Detecting a first input; In response to detecting the first input, ceasing to display the first text at the first location; and ceasing to display the second text at the second location; and and displaying the second text at the first location.

2. 10. The method of claim 1, displaying a handwriting input area via the display; and displaying, via the display, a candidate display area distinct from the handwriting input area, wherein the first text at the first position and the second text at the second position are displayed in the candidate display area.

3. The method of claim 2 , wherein the first input is detected in the handwriting input area.

4. The method of claim 2 , wherein the first text and the second text are determined based on handwriting input detected in the handwriting input area.

5. 5. The method of claim 1, wherein the second text is different from the first text.

6. The method of claim 1 , wherein the first input is detected at a location different from the first location and the second location.

7. 7. The method of claim 1, wherein the first input does not correspond to the first text.

8. 8. The method of claim 1, wherein the first input does not correspond to the second text.

9. A computer program product causing a computer to carry out the method according to any one of claims 1 to 8.

10. 1. An electronic device comprising: The display and one or more input devices; a memory for storing the computer program according to claim 9; and one or more processors capable of executing the computer programs stored in the memory.

11. 1. An electronic device including a display and one or more input devices, An electronic device comprising means for carrying out the method according to any one of claims 1 to 8.

Images