Processing text handwriting input in freehand mode

Abstract

The present invention relates to a method, the method comprising: detecting strokes of digital ink (IN) input on a computing device in a freehand writing format (FT1); detecting text blocks (BL1) from the strokes; performing text recognition on each text line of the text block, including extracting text lines from the text block (BL1) and generating model data that associates each stroke of the text block with characters, words, and text lines of the text block (BL1); normalizing each text line from the freehand writing format (FT1) to a structured format (FT2) to comply with a document pattern (200). The normalization may include, for each text line: calculating a transformation function to transform the text line into the structured format; applying the transformation function to the text line; and updating the model data of the text line based on the transformation function.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computing device interfaces capable of recognizing user input handwritten text. Specifically, the present disclosure relates to computing devices and corresponding methods for recognizing and editing handwritten text. Background Art

[0002] Computing devices are becoming increasingly common in daily life. They may take various forms, such as desktop computers, laptop computers, tablet computers, hybrid computers (2-in-1), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses / headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), gaming consoles, etc. In addition, computing devices are incorporated into vehicles and devices such as cars, trucks, farm equipment, manufacturing equipment, building environmental controls (e.g., lighting, HVAC), and household and commercial appliances.

[0003] Each type of computing device is equipped with specific computing resources and is used for a given purpose. Computing devices generally include at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. Various computing devices and their subsequent uses require various input devices and interfaces to allow users to interact with their computing devices.

[0004] One such input device is a touch-sensitive surface, such as a touch screen or touchpad, where user input is received through contact between a user body part (e.g., finger) or instrument (e.g., pen or stylus) and the touch surface. Another input device is an input surface that senses gestures made by the user above the input surface. Additional input devices are position detection systems that detect the relative position of touch or non-touch interactions with non-touch physical or virtual surfaces.

[0005] Handwriting recognition may be implemented in a computing device to input and process various types of input elements hand-drawn or handwritten by a user, such as text content (e.g., alphanumeric characters) or non-text content (e.g., shapes, diagrams). Once input on the computing device, the input elements are typically displayed as digital ink and undergo handwriting recognition to be converted into a typeset version. Real-time handwriting recognition systems or methods are generally used to interpret user handwritten input. For this purpose, online systems (recognition performed using cloud-based solutions, etc.) or offline systems may be used.

[0006] The user input can be a diagram or any other text, non-text content, or a mixture of text and non-text content. Handwriting input can be performed on a structured document according to guiding lines (or baselines) that guide and constrain the user input. Alternatively, the user can write in freehand mode, i.e., without any constraints on the lines to follow or the size of the input to adhere to (e.g., on a blank page).

[0007] Figure 1A An example of a computing device 1 including a display device 1 is shown, where the display device uses an appropriate user interface to display ink input elements hand-drawn or handwritten by the user in freehand mode. In this example, the computing device 1 detects and displays text content 4 and 6 as well as non-text content 8, 10, and 12. Each of these elements is formed by one or more strokes of digital ink. The input elements can include, for example, text handwriting, diagrams, musical notations, etc. In this example, the shape 8 is a rectangle or the like that forms a container (frame) enclosing the text content 6, such that elements 6 and 8 can be selected and manipulated together.

[0008] Handwriting recognition can be performed on text input elements and possibly also on non-text input elements. Additionally, each input element can be transformed and displayed as a typeset input element, as depicted in this example in Figure 1B .

[0009] In handwriting recognition applications, the performance in text recognition is not always satisfactory, especially in the case of text handwriting input in freehand mode (i.e., in a freehand format). The reliability and stability issues of the text recognition process often undermine the performance of such computing devices, thus limiting the overall user experience. The limitations may mainly stem from the difficulty of the computing device in determining when a new ink stroke affects the previous state of recognition (i.e., the previous recognition result based on the previous ink strokes of the input) and when it does not affect such a previous state of recognition (i.e., the new ink stroke is related to new content that does not affect the previous input content).

[0010] In addition, a certain level of editing can typically be performed on the user input displayed on the computing device. However, traditionally, such applications are limited in their ability to handle editing functions and often constrain the user to take actions that do not reflect the user's original intention or to accept compromises.

[0011] When inputting text by hand using a freehand writing mode, the display and editing functions are particularly restricted on a computing device. In the freehand writing mode, essentially no restrictions such as lines, size, orientation, margins, etc. are imposed on the user, allowing for the input of various complex handwritten forms, which makes it more difficult to operate on the handwritten text (e.g., move, re-scale, correct, insert line breaks in the text stream) by implementing an editing function on the computing device. However, users may wish to edit and operate on the handwritten input in a more structured and advanced manner, especially when using the freehand writing mode. These limitations in handling handwritten text input in the freehand format disrupt the user experience and require improvement.

[0012] A solution is needed that allows for the efficient and reliable handling of handwritten text input in a freehand writing mode (or format), especially improving text recognition and allowing for the efficient editing of such handwritten text on a computing device. Summary of the Invention

[0013] Examples of the present invention described below provide a computing device, a method, and a corresponding computing program for editing handwritten text input by a user.

[0014] According to a particular aspect, the present invention provides a method for processing handwritten text implemented by a computing device, including:

[0015] - Detecting a plurality of input strokes of digital ink through an input surface, the input strokes being input in a freehand writing format without any handwriting constraints;

[0016] - Displaying the plurality of input strokes on a display device in the freehand writing format;

[0017] - Classifying each input stroke as text or non-text, the classification including detecting at least one text block of handwritten text as text from the input strokes handwritten in the freehand writing format;

[0018] - Performing text recognition on the at least one text block, the text recognition including:

[0019] ○ Extracting text lines of the handwritten text from the at least one text block;

[0020] ○ Generating model data that associates each stroke of the at least one text block with characters, words, and text lines of the at least one text block;

[0021] - Normalizing each text line of the handwritten text from the freehand writing format to a structured handwritten format to comply with the line pattern of a document mode, the normalization including for each text line:

[0022] ○ Calculate a corresponding transformation function for the text line to transform the text line into a structured handwritten format;

[0023] ○ Apply the corresponding transformation function to transform each stroke of the text line into a structured handwritten format; and

[0024] ○ Update the model data of the text line based on the corresponding transformation function.

[0025] In a particular embodiment, the method includes storing the model data generated during the text recognition,

[0026] wherein updating the model data further includes storing the updated model data of the at least one text block to replace the model data generated during the text recognition.

[0027] In a particular embodiment, the method includes displaying the text line of the at least one text block in a structured handwritten format after the normalization.

[0028] In a particular embodiment, the model data of the at least one text block includes:

[0029] - Character information defining a plurality of characters, each character being associated with at least one stroke of digital ink and the text line of the at least one text block;

[0030] - Word information defining a plurality of words, each word being associated with at least one character defined by the character information; and

[0031] - Line information defining each text line of the at least one text block, each text line being associated with at least one word defined by the word information.

[0032] In a particular embodiment, for each text line of the at least one text block, the line information includes:

[0033] - Origin coordinates representing the origin of the text line;

[0034] - Inclination information representing the inclination of the text line; and

[0035] - Height information representing the height of the text line.

[0036] In a particular embodiment, updating the model data during the normalization includes updating the line information of the text line based on the corresponding transformation function.

[0037] In a particular embodiment, for each text line, the normalization includes:

[0038] - Determining input parameters, the input parameters including the origin coordinates, inclination information, and height information of the text line;

[0039] A corresponding transformation function is calculated based on the input parameters and the document mode.

[0040] In a particular embodiment, the document mode defines at least one of the following handwriting constraints to be adhered to by the handwritten text:

[0041] - The margin of the display area; and

[0042] - The line spacing.

[0043] In a particular embodiment, each transformation function defines at least one of the following transformation components to be applied to the corresponding text line during the normalization:

[0044] - The translation component;

[0045] - The scaling component; and

[0046] - The rotation component.

[0047] In a particular embodiment, the document mode includes the line mode that defines the guide lines, and the handwritten text will be arranged in a structured handwritten format according to the line mode.

[0048] In a particular embodiment, the ratio of the distance between two consecutive guide lines based on the line mode to the height of the corresponding text line determines the scaling component of the transformation function during the normalization.

[0049] In a particular embodiment, the translation component of the transformation function is determined during the normalization to perform the translation of the text line, so that the origin of the text line is moved to align with the corresponding guide line of the line mode, and the corresponding guide line is assigned to the text line during the normalization.

[0050] In a particular embodiment, the rotation component is determined during the normalization to rotate the corresponding text line to reduce its inclination to zero according to the document mode.

[0051] In a particular embodiment, during the normalization, the model data of each text line is updated according to the corresponding transformation function, while preventing any text recognition that may be caused by the application of the corresponding transformation function.

[0052] According to another aspect, the present invention relates to a non-transitory computer-readable medium having computer-readable program code (or computer program) recorded thereon, and the computer-readable program code includes instructions for performing the steps of the method of the present invention defined in this document.

[0053] The computer program of the present invention can be expressed in any programming language and can be in the form of source code, object code, or any intermediate code between source code and object code, such as in a partially compiled form, or in any other suitable form.

[0054] The present invention also provides a computer program as described above.

[0055] The previously mentioned non-transitory computer-readable medium can be any entity or device capable of storing a computer program. For example, the recording medium can include storage components such as ROM memory (CD-ROM or ROM implemented in a microelectronic circuit), or magnetic storage components such as a floppy disk or a hard disk, etc.

[0056] The non-transitory computer-readable medium of the present invention can correspond to a transmissible medium, such as an electrical signal or an optical signal, which can be transmitted via a cable or an optical fiber cable or by radio or any other suitable means. The computer program according to the present disclosure can specifically be downloaded from the Internet or a similar network.

[0057] Alternatively, the non-transitory computer-readable medium can correspond to an integrated circuit in which a computer program is loaded, and the circuit is suitable for executing or used for executing the method of the present invention.

[0058] In a specific embodiment, the present invention relates to a non-transitory computer-readable medium having computer-readable program code embodied therein, the computer-readable program code being suitable for execution to implement a method for hand-drawing input elements on a computing device, the computing device including a processor for performing the steps of the method.

[0059] The present invention also relates to a computing device suitable for implementing the method defined in the present disclosure. More specifically, the present invention provides a computing device for handwritten text, including:

[0060] - An input surface for detecting a plurality of strokes of digital ink, the strokes being input in a freehand writing format without any handwriting constraints;

[0061] - A display device for displaying the plurality of input strokes in the freehand writing format;

[0062] - A classifier for classifying each stroke as text or non-text, the classifier being configured to detect at least one text block of handwritten text as text from the input strokes handwritten in the freehand writing format;

[0063] - A line extractor for extracting text lines of handwritten text from the at least one text block;

[0064] - An identification engine for performing text recognition on each text line of the at least one text block, thereby generating model data associating each stroke of the at least one text block with characters, words, and text lines of the at least one text block;

[0065] - A text editor for normalizing each text line of handwritten text from a freehand format to a structured handwritten format to comply with the line pattern of a document pattern, the text editor being configured to perform, for each text line:

[0066] ○ Computing a corresponding transformation function for the text line to transform the text line into a structured handwritten format;

[0067] ○ Applying the corresponding transformation function to transform each stroke of the text line into a structured handwritten format; and

[0068] ○ Updating the model data of the text line based on the corresponding transformation function.

[0069] The various embodiments defined above in connection with the method of the present invention are applied in a similar manner to the computing device, computer program, and non - transitory computer - readable medium of the present disclosure.

[0070] For each step of the method of the present invention defined in the present disclosure, the computing device may include a corresponding module configured to perform the step.

[0071] In a particular embodiment, the present disclosure may be implemented using software and / or hardware components. In this context, the term "module" in the present disclosure may refer to a software component, a hardware component, or multiple software and / or hardware components. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Other features and advantages of the present disclosure will be apparent from the following description with reference to the accompanying drawings, which show embodiments without limiting features. In the drawings:

[0073] - Figures 1A to 1B Represents a digital device according to a conventional arrangement;

[0074] - Figure 2 Is a block diagram schematically representing a computing device according to a particular embodiment of the present invention;

[0075] - Figure 3 Schematically depicts a normalization process according to a particular embodiment of the present invention;

[0076] - Figure 4 Is a block diagram schematically representing a module implemented by a Figure 2 computing device according to a particular embodiment of the present invention;

[0077] -Figure 5 is a flowchart schematically showing the steps of a method according to a particular embodiment of the present invention;

[0078] - Figures 6 to 8 is a schematic representation of text handwriting when processed by a Figure 2 computing device according to a particular embodiment of the present invention;

[0079] - Figure 9 schematically depicts model data generated by a Figure 2 computing device according to a particular embodiment of the present invention;

[0080] - Figure 10 schematically shows text handwriting that has been normalized by a Figure 2 computing device according to a particular embodiment of the present invention;

[0081] - Figure 11 represents editing operations applied to text handwriting during the normalization process according to a particular embodiment of the present invention; and

[0082] - Figure 12 schematically shows text handwriting edited after normalization according to a particular embodiment of the present invention.

[0083] Components in the diagrams are not necessarily drawn to scale, but emphasis is placed on illustrating the principles of the present invention.

[0084] For simplicity and clarity of illustration, the same reference numerals will be used throughout the diagrams to refer to the same or like parts, unless otherwise indicated. Detailed Description

[0085] In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, those skilled in the art will appreciate that the teachings may be practiced without such details. In other instances, well-known methods, procedures, and / or components are described at a relatively high level without detailed description so as not to unnecessarily obscure aspects of the teachings.

[0086] The following description of exemplary embodiments refers to the accompanying drawings. The following detailed description does not limit the present invention. Instead, the scope of the present invention is defined by the appended claims. In the various embodiments illustrated in the diagrams, computing devices, corresponding methods, and corresponding computer programs are discussed.

[0087] The use of the term "text" in this specification is understood to cover all characters (e.g., alphanumeric characters, etc.) and strings thereof in any written language, as well as any symbols used in written text.

[0088] The term "non-text" in this specification is understood to cover free-form handwritten or hand-drawn content (e.g., shapes, drawings, etc.) and image data, as well as characters and strings thereof, or symbols used in non-text contexts. Non-text content defines graphical or geometric structures in linear or non-linear configurations, including containers, drawings, common shapes (e.g., arrows, blocks, etc.). For example, in a diagram, text content may be contained within a shape (rectangle, oval, ovoid, …) called a container.

[0089] In addition, the examples shown in these figures are in left-to-right written language text, and thus any reference to position can apply to written language with different orientation formats.

[0090] The various techniques described herein generally relate to capturing, processing, and managing handwritten or hand-drawn content on portable and non-portable computing devices. The systems and methods described herein can utilize the recognition of a user's natural writing and drawing style, which is input into the computing device via an input surface such as, for example, a touch screen (as discussed later). Although the various embodiments are described with respect to recognizing digital ink handwritten input using so-called online recognition techniques, it should be understood that application to other forms of input for recognition (e.g., offline recognition) is possible, such as involving a remote device or server for performing the recognition.

[0091] The terms "hand-drawn" and "handwritten" are used interchangeably herein to define the creation of digital content (handwritten input) by a user using their hand (or finger) or an input device (handheld stylus or digital pen, mouse, …) on or using an input surface. The term "hand" etc. is used herein to provide a concise description of the input technique, but this definition includes the use of other parts of the user's body (e.g., feet, mouth, and eyes) for similar input.

[0092] As described in more detail below, aspects of the present invention rely on detecting text handwriting of digital ink input to a computing device in a freehand format; performing text recognition, which involves generating model data representing the text handwriting input (or input text handwriting); and performing normalization of each text line in the text handwriting input from a freehand format to a structured handwriting format to comply with a document mode (e.g., comply with the line mode of the document mode). Such normalization means performing edit transforms (e.g., translation, rotation, and / or rescaling) on the text handwriting input to convert it into a structured handwriting format. For the purpose of normalization, the model data is also updated according to the edit transforms applied to the text handwriting input. In other words, the transform function applied to each corresponding text line in the text handwriting input when performing normalization also serves as the basis for updating the model data of the text line. As explained in a specific embodiment below, the model data defines the correlation between each stroke of the text handwriting input and the corresponding characters, corresponding words, and corresponding text lines of the text handwriting input.

[0093] By updating the model data as part of the normalization process, the text handwriting input of digital ink can be converted from an unconstrained environment (in freehand format) to a formatted environment (in structured handwriting format), thereby enabling better display, more reliable text recognition, and more extensive operations, such as editing the text handwriting input. As further explained below, although the normalization process results in modification of the input ink, any previous text recognition state is retained. This can be achieved because the text handwriting input and the model data are updated using the same transform.

[0094] In the present invention, each text line undergoing normalization is a text line of handwritten text. As further described below, although some transforms are applied to this handwritten input line as part of the normalization process, these text lines remain handwritten text lines once normalized (i.e., normalized text lines), that is, handwritten in a normalized manner as text lines (e.g., as opposed to typeset content that does not constitute handwriting formed by input strokes). As further described below, a transform function is applied to the strokes of each text line of the handwritten text, and the transform function defines the transform for converting the corresponding handwritten text line into a normalized handwritten text line.

[0095] Figure 2A block diagram of a computing device 100 is shown in accordance with a particular embodiment of the present invention. The computing device (or digital device) 100 can be a desktop computer, laptop computer, tablet computer, e - book reader, mobile phone, smartphone, wearable computer, digital watch, interactive whiteboard, global positioning system (GPS) unit, enterprise digital assistant (EDA), personal digital assistant (PDA), gaming console, etc. The computing device 100 includes components of at least one processing element, some form of memory, and input and output (I / O) devices. The components communicate with each other through input and output, such as connectors, lines, buses, links, networks, or other input and output known to those skilled in the art.

[0096] More specifically, the computing device 100 includes an input surface 104 for hand - drawn (or handwritten) input elements, which include text and non - text elements, as further described below. More specifically, the input surface 104 is adapted to detect a plurality of input strokes of digital ink input on the input surface. As further discussed below, these input strokes can be input in a free - handwritten format (or in a free - handwritten mode), i.e., with no handwriting constraints on position, size, and direction in the input area.

[0097] The input surface 104 can employ technologies such as resistive, surface acoustic wave, capacitive, infrared grid, infrared acrylic projection, optical imaging, dispersive signal technology, acoustic pulse recognition, or any other suitable technology known to those skilled in the art to receive user input in the form of a touch - sensitive or proximity - sensitive surface. The input surface 104 can be a non - touch - sensitive surface monitored by a position - detection system.

[0098] The computing device 100 further includes at least one display device (or display) 102 for outputting data such as images, text, and video from the computing device. The display device 102 can be a screen or the like of any suitable technology (LCD, plasma, etc.). As further described below, the display device 102 is adapted to display input elements in digital ink, each input element being formed by at least one stroke of digital ink. Specifically, the display device 102 can display, for example, the plurality of strokes input through the input surface 104 in the above - mentioned free - handwritten format.

[0099] The input surface 104 can be in the same position as the display device 102 or remotely connected to the display device. In a particular instance, the display device 102 and the input surface 104 are part of a touch screen.

[0100] As Figure 2 depicted, the computing device 100 further includes a processor 106 and a memory 108. The computing device 100 can also include one or more volatile storage elements (RAM) as part of or separate from the memory 108.

[0101] Processor 106 is a hardware device for executing software, specifically software stored in memory 108. Processor 108 can be any custom or commercially available general-purpose processor, central processing unit (CPU), semiconductor-based microprocessor (in the form of a microchip or chipset), microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, or any combination thereof, and more generally, any suitable processor component designed to execute software instructions known to those skilled in the art.

[0102] According to a particular embodiment of the present disclosure, memory 108 constitutes (or includes) a non-transitory (or non-volatile) computer-readable medium (or recording medium). Memory 108 can include any combination of non-volatile storage elements (such as ROM, EPROM, flash PROM, EEPROM, hard disk drive, magnetic tape or optical tape, memory register, CD-ROM, WORM, DVD, etc.).

[0103] Memory 108 can be remote from computing device 100, such as at a server or cloud-based system that can be remotely accessed by computing device 100. Non-volatile memory 108 is coupled to processor 106 such that processor 106 can read information from and write information to memory 108. As an alternative, memory 108 is integral with computing device 100.

[0104] Memory 108 includes an operating system (OS) 110 and a handwriting application (or computer program) 112. Operating system 110 controls the execution of application 112. According to a particular embodiment of the present invention, application 112 constitutes (or includes) a computer program (or computer-readable program code) that includes instructions for implementing the method according to a particular embodiment of the present invention.

[0105] Application 112 can include instructions for detecting and managing ink input elements handwritten by a user on input surface 104 of computing device 100. As discussed later, these handwritten ink input elements (also referred to as handwritten input), which can be formed by one or more strokes of digital ink, can be text or non-text.

[0106] Application 112 can include a handwriting recognition (HWR) module (or system) 114 for recognizing handwritten input (including handwritten text and non-text) of computing device 100. HWR 114 can be a source program, executable program (object code), script, application, or any other component having an instruction set to be executed. In Figure 2In the present example depicted, the application 112 and the HWR module 114 are combined in a single application (the HWR module 114 is part of the application 112). Alternatively, the HWR module 114 can be a module, method, or system for communicating with a handwritten recognition system remote from the computing device 100, such as the server (or cloud-based system) SV1 depicted in Figure 2 The application 112 and the HWR module 114 can also be separate components stored in the memory 108 (or a different memory) of the computing device 100, whereby the application 112 and the HWR module 114 operate together to access information processed and stored in the memory 108.

[0107] As shown later in the figures, input strokes entered on or via the input surface 104 are processed by the processor 106 as digital ink. In this case, digital ink is formed by presenting the handwritten input in digital image format on the display device 102.

[0108] The user can enter input strokes with the hand or finger, or with some input tool, such as a digital pen or stylus suitable for use with the input surface 104. If components configured to sense movement near the input surface 104 are used, or via a peripheral device of the computing device 100, such as a mouse or joystick, etc., the user can also enter input strokes by making gestures above the input surface 104.

[0109] Each ink input element (letter, symbol, word, shape, …) is formed by one or more such input strokes or at least by a portion of a stroke. A stroke (or input stroke) is characterized at least by a stroke start position (corresponding to a “pen down” event), a stroke end position (corresponding to a “pen up” event), and a path connecting the stroke start position to the stroke end position. Since different users may naturally write or hand-draw the same object (e.g., letter, shape, symbol, …) but with slight variations, the HWR module 114 provides multiple ways in which each object can be entered while still being recognized as the correct or expected object.

[0110] The handwriting application 112 allows the generation of handwritten or hand-drawn content (e.g., text, diagrams, charts, shapes, drawings, or any kind of text and / or non-text handwritten input) in the form of digital ink and accurately recognizes this content using the HWR module 114.

[0111] In this example, the memory 108 is also suitable for storing model data DT that defines a model document representing a text handwritten input from a user on the computing device 102 having the input surface 104. The nature and use of this model data DT will be discussed in more detail below.

[0112] As Figure 3 shown, in this embodiment, the computing device 100 is configured to detect a handwritten input IN entered in a freehand format (or freehand mode) FT1, i.e., through the input surface 104 without any handwriting constraints on the user. The freehand mode allows the user to handwrite input elements IN in a free environment (e.g., in the blank area Z1) in an unstructured or unguided manner, i.e., without any handwriting constraints on the position, size, and orientation of the text handwritten input (no line pattern to follow, no size or orientation limitations, no limitations on inter-line, margins, etc.). This freehand mode FT1 provides complete freedom to the user during handwritten input, which is sometimes required, for example, to record quick and miscellaneous notes or to perform a mixed input of text and non-text.

[0113] The handwritten input IN is formed by a plurality of input strokes of digital ink, which is detected by the input surface 104 and displayed by the display device 102.

[0114] As Figure 3 depicted, the computing device 100 can classify the input strokes forming the handwritten input IN as text and thus recognize the handwritten input IN as a text block BL1.

[0115] The computing device 100 is configured to perform a normalization process on the recognized text block BL1, thereby converting the text block BL1 from the freehand format FT1 to a structured handwritten format (or structured handwritten mode) FT2 (also referred to as a structured format (or structured mode)) to comply with the document mode 200 (e.g., comply with the line pattern of the document mode 200). The document mode is understood in this disclosure to define geometric constraints regarding how to arrange and display the text handwritten input IN on the display device 102. The document mode 200 constitutes a structured (or formatted) environment to receive the handwritten input IN (in this case, the text block BL1) in a structured manner, i.e., in a structured handwritten format FT2.

[0116] It should be noted that the document mode 200 defines how to arrange handwriting structurally.

[0117] The document mode may define at least one of the following handwriting constraints that the text will comply with:

[0118] - The outer margins of the display area;

[0119] - The line pattern; and

[0120] - Line spacing.

[0121] Specifically, the document mode 200 may define a line mode and may also define at least one of the margins and the line spacing of the display area.

[0122] In Figure 3 the present example shown, the document mode 200 includes a line mode that defines a plurality of guide lines (also referred to as guide lines or baselines) 202, according to which the handwritten text (i.e., the content of the text block BL1) will be arranged according to the structured handwriting format FT2. The document mode 202 may define a predetermined line spacing d1 between each pair of consecutive guide lines, thereby imposing a size constraint on the handwritten text in the structured handwriting format FT2. The document mode 202 may also define a predetermined line length d2, which imposes a maximum length on each text line of the text block BL1 in the structured format FT2. It should be understood that this structured format FT2 based on the document mode 200 only constitutes an example of an embodiment. Those skilled in the art may conceive of other document modes.

[0123] As shown according to a particular embodiment in Figure 4 when running the application program 112 stored in the memory 108 ( Figure 2 ), the processor 106 implements a plurality of processing modules, namely: a classifier MD2, a line extractor MD4, an identification engine MD6, and an editing module MD8. The application program 112 includes instructions for configuring the processor to implement these modules in order to execute the method steps of the present invention, as described later in a particular embodiment.

[0124] The classifier MD2 is configured to classify each input stroke (or any set or combination of input strokes) detected by the input surface 104 as text or non-text. Specifically, as further described below, the classifier MD2 may be configured to detect at least one text block BL1 of the input strokes input in the freehand format FT1 through the input surface 104 as text.

[0125] To this end, the classifier (or classification module) MD2 may perform a disambiguation process to distinguish text from non-text content in the input digital ink input by the user. The disambiguation process may be performed in any manner known to those skilled in the art. Exemplary embodiments are described in U.S. Patent Application No. 2017 / 0109578A1.

[0126] As an example of the disambiguation process, classifier MD2 can use spatial and temporal considerations to group strokes to build hypotheses about which strokes may belong to non-text or text elements. Spatial considerations can include the distance between strokes, the geometry of the strokes, the overlap of the strokes, and the relative position of the strokes. Temporal considerations can include the chronological order of stroke input. Probability scores can be calculated such that only hypotheses with a sufficiently high score (above a threshold) are retained. Then, features of each group of strokes are extracted considering shape and text language models. Features can include inter-character spacing, direction changes within a stroke, overlap, direction of the stroke pattern, and curvature. Then, classifier MD2 can classify the strokes into text and non-text by testing the hypotheses based on all the information collected, including the extracted features of the stroke groups within these hypotheses and the spatial and temporal information of the strokes within these groups. Of course, as already mentioned, other techniques for non-text / text discrimination can be envisioned.

[0127] In other instances, classification as text / non-text is not required because computing device 100 receives only text as handwritten input such that text handwritten is directly detected through input surface 104.

[0128] A line extractor (or line extraction module) MD4 is configured to extract text lines LN from a text block BL1 of input strokes detected by classifier MD2. In other words, line extractor MD4 is capable of identifying the different text lines LN that together form text block BL1 (text block BL1 is thus divided into multiple text lines LN). As further described below, such line extraction can be performed as part of the text recognition process. In a particular instance, line extractor MD4 and recognition engine MD6 together form a single module.

[0129] A recognition engine (or recognition module) MD6 is configured to perform text recognition on each text line LN of text block BL1 extracted by classifier MD2. Any suitable techniques known to those skilled in the art can be used to perform text recognition, such as those involving generating a list of element candidates (or hypotheses) with probability scores and applying a language model (dictionary, grammar, semantics, …) to the element candidates to find the best recognition result. Statistical information modeling the frequency with which a given sequence of elements occurs in a specified language or is used by a particular user can also be considered to evaluate the likelihood of the interpretation results produced by recognition engine MD6. However, no further details are provided to avoid unnecessarily obscuring the present disclosure. Examples of implementing handwritten recognition can be found, for example, in U.S. Patent Application No. 2017 / 0109578A1.

[0130] In the present embodiment, when performing text recognition, the recognition engine MD6 is configured to generate model data DT that associates each input stroke of text block BL1 with a unique character, unique word, and unique text line of text block BL1. As described further below, the model data DT defines the correlation (or link or reference) between each stroke of the text handwritten input and the corresponding character, corresponding word, and corresponding text line of the text handwritten input.

[0131] The editing module (or text editor) MD8 is configured to normalize each text line LN from the above freehand format FT1 to a structured format FT2 to comply with the document mode 200 (as Figure 3 illustrated). As described further below, during this normalization, the editing module MD8 can be configured to perform, for each text line LN:

[0132] - Calculate a corresponding transformation function for the text line LN to transform the text line LN into the structured format FT2 to comply with the document mode 200 (e.g., comply with the line mode of the document mode 200);

[0133] - Apply the corresponding transformation function to transform each input stroke of the text line LN into the structured format FT2; and

[0134] - Update the model data DT (and more specifically, the model data DT associated with the text line LN) based on the corresponding transformation function.

[0135] In other words, when normalizing the text block BL1, the editing module BL1 converts the text block BL1 from the freehand format FT1 to the structured handwritten format FT2 line by line. Once normalized by the editing module BL1, the normalized content remains essentially handwritten (formed by input strokes), even if some transformations are applied to the strokes forming the handwritten as part of the normalization process.

[0136] Once the normalization has been performed, the display device 102 can be configured to display the text block BL1 in the structured format FT2. The editing module MD8 can also be configured to perform further editing on the text block BL1 in the structured format FT2.

[0137] The configuration and operation of the modules MD2 to MD8 of the computing device 100 will be more apparent in the specific embodiments described below with reference to the drawings. It should be understood that the modules MD2 to MD6 as Figure 4 illustrated only represent example embodiments of the present invention, and other embodiments are possible.

[0138] For each step of the method of the present invention, the computing device may include corresponding modules configured to perform said steps. At least two of these steps may be performed as part of a single module.

[0139] According to a particular embodiment of the present invention, reference is now made to Figures 5 to 12 a method implemented by a computing device 100 as illustrated in Figures 2 to 4 . More specifically, the computing device 100 implements this method by executing an application program 112 stored in a memory 108.

[0140] Consider an example scenario where a user inputs a handwritten input IN as shown in Figure 6 on the computing device 100. Then processing is performed by the computing device 100, including a normalization process for the handwritten input IN as described below.

[0141] More specifically, in a detection step S2, the computing device 100 detects the handwritten input IN input by the user through an input surface 104 of the computing device 100. As shown in Figure 6 , the handwritten input IN includes a plurality of input strokes ST of digital ink formed by the user through the input surface 104. For example, a first string of the character “This” is formed by the input strokes ST1, ST2, ST3, and ST4. As already indicated, each input stroke ST is characterized by at least a stroke start position, a stroke end position, and a path connecting the stroke start position and the stroke end position. Thus, for example, a point located at the top of the character “i” (in the word “This”) itself constitutes a single stroke ST4.

[0142] In this example, the handwritten digital ink IN is input in the freehand format FT1 as previously described ( Figure 3 ), i.e., without any handwriting constraints in a predetermined input area of the display 102. Without any constraints such as lines, sizes, orientations, etc. to comply with, the user is allowed to handwrite the content IN in a free and easy manner. As can be seen, the size, orientation, and position of each handwritten character or each handwritten word can vary arbitrarily according to the user's preference. Although the handwritten input IN is shown in Figure 6 along substantially parallel lines, the handwritten input IN can be arranged in different line orientations.

[0143] As shown in Figure 7 , the computing device 100 displays (S4) the plurality of input strokes ST of the handwritten input IN on the display device 102 according to the freehand format (or mode) FT1.

[0144] In classification step S6, computing device 100 classifies each input stroke ST detected in freehand format FT1 as text or non - text. For this purpose, classifier MD6 of computing device 100 can perform the disambiguation process as previously described in any suitable manner.

[0145] For simplicity, in this example, it is assumed that the entire handwritten input IN detected by computing device 100 is text (i.e., handwritten text). Thus, between classification steps S6, computing device 100 detects text block BL1 of the handwritten text formed by input strokes ST entered in freehand format FT1 as text. It should be noted that for simplicity, in this example, computing device 100 detects a single text block BL1 of handwritten text, but the concept of the present invention will apply in the same way to multiple text blocks of handwritten text detected by computing device 100 in handwritten input IN.

[0146] As Figure 8 shown, computing device 100 then performs (S8) text recognition on text block BL1. During text recognition S8, computing device 100 analyzes the individual features of input strokes ST to identify predefined recognizable text elements, such as characters CH (or symbols) and words WD ( Figure 8 ). As already mentioned, any suitable technique known to those skilled in the art can be used to perform text recognition, including for example generating a list of element candidates (or hypotheses) with probability scores and applying a language model (dictionary, grammar, semantics, …) to the element candidates to select the most suitable candidate. For this purpose, recognition engine MD6 can use any appropriate semantic information to parse and analyze the content of each text line LN to identify predefined characters.

[0147] Text recognition S8 includes two steps described below, namely, line extraction step S10 and generation step S12.

[0148] More specifically, during text recognition S8, computing device 100 performs line extraction S10 to extract text lines LN of the handwritten text from text block BL1. In this example, computing device 100 divides text block BL1 into 5 different text lines LN1 to LN5. Based on the geometric analysis of strokes ST (or multiple sets of strokes ST) of text block BL1 detected during text / non - text classification S6, line extraction can be performed by line extractor MD4. Based on text analysis, line extractor MD4 is able to determine which text line each input stroke ST belongs to. Those skilled in the art can implement any suitable technique to allow the identification of text lines within a text block entered in freehand mode.

[0149] Still during text recognition S8, computing device 100 also generates (S12) model data DT that associates each stroke ST of text block BL1 with a unique character CH, a unique word WD, and a unique text line LN of text block BL1.

[0150] Model data DT defines the document model (also referred to as the interactive text model) of text block BL1. Each text line LN is structured according to this document model. As shown later, by establishing these correlations, ink interactivity can be achieved when editing text block BL1 later. For example, erasing word WD in text block BL1 causes computing device 100 to also erase all constituent characters CH referenced for this word WD according to model data DT.

[0151] In this example, model data DT includes three categories: character information, word information, and line information. Model data, which can be organized in any suitable way (e.g., as a relational tree or a reference tree), defines the cross-reference that links each stroke ST (or each part of a stroke) to a unique character CH, each character CH to a unique word WD, and each word WD to a unique text line LN, thereby establishing the correlation between stroke ST (or part of a stroke), character CH, word WD, and text line LN as detected in text recognition S8. Thus, digital ink is structured and associated with model data DT to form interactive ink.

[0152] More specifically, according to a particular example, as Figure 9 shown, the model data DT of text block BL1 can include the following:

[0153] - Character information 220 that defines a plurality of characters CH, each character CH being associated with at least one stroke ST (or at least a part of stroke ST) of digital ink and a text line LN of text block BL1;

[0154] - Word information 222 that defines a plurality of words WD, each word WD being associated with at least one character CH defined by character information 220; and

[0155] - Line information 224 that defines each text line LN of text block BL1, each text line LN being associated with at least one word WD (or a part of a word) defined by word information 222.

[0156] It should be noted that a character CH can be formed by a single input stroke ST, multiple input strokes ST, or a part of one or more input strokes ST. Character information 220 represents the link of each part of stroke ST to a unique character CH. For example, as Figure 6As shown, the characters CH2, CH3, and CH4 recognized by the computing device 100 during text recognition S10 are formed, in part or in whole, by the portions ST31, ST32, and ST33 of the stroke ST3, respectively.

[0157] In a particular instance, for each text line LN of the text block BL1, the line information 224 includes:

[0158] - Origin coordinates (x, y) representing the origin of the text line LN;

[0159] - Inclination information (a) representing the inclination of the text line LN; and

[0160] - Height information (h) representing the height of the text line LN.

[0161] Thus, the line information 224 allows the computing device 100 to define the position, orientation, and size of each text line LN.

[0162] It should be noted that any predetermined reference point belonging to the text line LN is defined as the above-mentioned "origin" of the text line LN in the line information 224.

[0163] The origin coordinates of the text line LN can be defined by a pair of floating-point values: Line(x, y). The inclination information can be a floating-point value: Line(a). The height information can be a floating-point value: Line(h), which represents, for example, the average height of the characters assigned to this text line according to the document model. In this instance, for each given text line LN, the height (h) is represented orthogonally to the inclination (a).

[0164] The computing device 100 can perform the calculation of the average height (h) of each text line LN such that the ascenders and descenders of the alphabet are taken into account to avoid interference with the resulting average value for each text line.

[0165] By way of example, the computing device 100 recognizes (S10) the input strokes ST1 to ST4 in the first text line LN1 as jointly forming the word WD1 in a particular structured manner. Thus, as Figure 6 and Figure 9 shown, the computing device 100 generates (S12) model data DT, which includes character information 220 defining the following characters associated with the text line LN1:

[0166] - Character CH1 associated with the stroke ST1 and the stroke ST2;

[0167] - Character CH2 associated with the portion ST31 of the stroke ST3;

[0168] - The character CH3 associated with a part ST32 of the stroke ST3 (supplementary parts ST31 and ST33) and the stroke ST4; and

[0169] - The character CH4 associated with a part ST33 of the stroke ST3 (supplementary to ST31 and ST32).

[0170] Other characters CH in the text block BL1 are defined in a similar manner in the character information 220.

[0171] Still in this example, the word information 222 defines the word WD1 associated with the characters CH1, CH2, CH3, and CH4. Other words WD in the text block BL1 are defined in a similar manner in the word information 222.

[0172] Still in this example, the line information 224 defines the following lines:

[0173] - The text line LN1, which shows the origin coordinates (x1, y1), the inclination information (a1), and the height information (h1);

[0174] - The text line LN2, which shows the origin coordinates (x2, y2), the inclination information (a2), and the height information (h2); etc.

[0175] Other text lines LN3, LN4, and LN5 of the text block BL1 are defined in a similar manner in the line information 224.

[0176] However, it should be understood that the model data DT can be organized in a manner different from Figure 9 the manner shown.

[0177] In the storage step S14, the computing device 100 stores the model data DT generated in S12 in the memory 108.

[0178] As Figure 10 shown, in the normalization step S16, the computing device 100 normalizes each handwritten text line LN of the text block BL1 from the freehand writing format FT1 to the structured handwritten format FT2 as previously described. This normalization is performed line by line as described in a specific example below. This normalization allows the computing device 100 to convert the text block BL1 of handwritten text from an unconstrained environment (freehand writing mode FT1) to a formatted or structured environment (structured handwritten format FT2), such that the handwritten text can be arranged and later displayed in a more organized (i.e., normalized) and efficient manner according to the document mode 200.

[0179] In Figure 10In an example, the result of the normalization S16 is displayed on the display device 102 by the computing device 100. As can be seen, according to the structured format FT2 defined by the document mode 200, the text lines LN1 to LN5 of the handwritten text are normalized into a uniform arrangement of position, size, and orientation. For example, the text lines LN2 and LN4 are reduced in size to comply with the line spacing d1 imposed by the document mode 200 (more specifically, by the line mode). All the text lines LN of the handwritten text are aligned and arranged according to the guide line 202. As has been indicated, various other forms of normalization can be envisioned within the present invention.

[0180] This normalization S16 is achieved by transforming (S20) each text line LN of the text block BL1 such that the text lines are rearranged according to the structured handwritten format FT2. As described further below, these transformations are performed by applying the corresponding transformation function TF to each text line LN (i.e., the strokes ST of each text line LN) of the handwritten text.

[0181] As already mentioned, each text line LN that undergoes normalization is a text line of the handwritten text. Although some transformations are applied to this handwritten input line as part of the normalization process S16, these text lines LN, once normalized (i.e., normalized text lines), remain handwritten text lines, i.e., handwritten (e.g., as opposed to typeset content that does not form the handwritten made up of input strokes) as text lines but arranged in a normalized manner. As described further below, the transformation function TF is applied to the strokes ST of each text line LN of the handwritten text, and the transformation function defines the transformation for converting the corresponding handwritten text line into a normalized handwritten text line.

[0182] In addition, the computing device 100 updates (S22) the model data DT representing the handwritten text line LN based on the same transformations (as defined by the transformation function) applied to the text line LN during the normalization S16.

[0183] More specifically, during the normalization step S16 ( Figure 5 ), the computing device 100 performs the same iteration for each text line LN of the text block BL1, including steps S18, S20, and S22. For the sake of clarity, the iteration S18 to S22 is described in detail below only with respect to the first text line LN1. Steps S18 to S22 can be applied in a similar manner for each text line LN of the text block BL1.

[0184] In determination step S18, computing device 100 computes (or determines) a corresponding transformation function TF to transform text line LN1 of the handwritten text into a structured handwritten format FT2 to conform to document mode 200. In a particular instance, computing device 100 obtains input parameters in S18, including line information 224 of text line LN1, i.e., the origin coordinates (x1, y1), inclination information (a1), and height information (h1) of text line LN1 of the handwritten text. Computing device 100 then determines (S18) the transformation function TF of text line LN1 based on the obtained input parameters of the handwritten text and based on document mode 200 (e.g., considering the line mode of document mode 200 to which structured format FT2 can be applied). Specifically, computing device 100 can determine transformation function TF such that the handwritten text of text line LN1: moves to the first baseline 202 of document mode 200, is oriented according to this first baseline, and is rescaled according to the line spacing of document mode 200. For example, the origin of text line LN1 is moved to a predetermined position on the first baseline 202.

[0185] As depicted in Figure 11 each transformation function TF computed by computing device 100 for a corresponding text line LN can define at least one of the following transformation components to be applied to the text line LN of the handwritten text (and thus to the strokes ST forming this text line LN) during normalization step S16:

[0186] - Translation component CP1;

[0187] - Scaling component CP2; and

[0188] - Rotation component CP3.

[0189] In the current case, each transformation function TF is considered to include these 3 components CP1, CP2, and CP3, but other embodiments are possible.

[0190] Translation component CP1 can define the translation of the corresponding text line LN. Scaling component CP2 can define the scaling operation on the corresponding text line. Finally, rotation component CP3 can define the rotation of the corresponding text line LN.

[0191] In a particular instance, during normalization step S16, the translation component CP1 of transformation function TF is determined to perform the translation of the text line (e.g., LN1) such that the origin of this text line is moved to align with the corresponding baseline 202, which is assigned to the text line during normalization step S16.

[0192] In a particular instance, during the normalization step S16, the scaling component CP2 of the transformation function TF is determined for a corresponding text line (e.g., LN1) based on the ratio of the distance d1 between two consecutive baselines 202 of the line pattern of the document pattern 200 with respect to the height of the text line (e.g., h1).

[0193] In a particular instance, during the normalization step S16, the rotation component CP3 is determined as rotating the corresponding text line (e.g., LN3) to reduce its inclination (e.g., a1) to zero according to the document pattern 200 (e.g., referring to the baseline 202 of the document pattern 200).

[0194] By way of example, the transformation function TF can define the rotation applied to the text line LN during normalization. The rotation applied to the inclined line LN can be a rotation by the opposite angle of the inclination (a). For a given inclination (a) of the text line LN, the rotation component CP3 of the transformation TF can thus be a rotation of (-a). When the transformation function TF implies a rotation, it may cause each point of the affected stroke or stroke part to rotate around the origin of the text line LN. As further explained below, the line information 224 can also be updated accordingly (by setting the line inclination (a) to zero: a = 0).

[0195] It should be noted that the rotation of the text line LN does not always require converting the text line into the structured format FT2, that is, when the text line LN is already in the appropriate orientation (e.g., when the text line LN is handwritten in the freehand format FT1 substantially along the direction of the corresponding baseline 202 of the document pattern 200). In such a case, the rotation component CP3 can be set such that CP3 = 0.

[0196] Similarly, in some cases, if translation and / or rescaling are not required respectively to convert the corresponding text line LN into the structured format FT2, the components CP1 and / or CP2 can be set to zero.

[0197] In a particular instance, the computing device 100 determines in S18 the transformation function TF of the text line LN1 such that at least one of the components CP1, CP2, and CP3 is non - zero. In the Figure 11 instance shown, the transformation function TF is computed such that the text line LN1 read as “Thisexample shows” written by hand is moved (translated) on the display device 102 according to the translation component CP1, rescaled (shrunk) according to the scaling component CP2, and rotated by a certain angle according to the rotation component CP3.

[0198] In transformation step S20, computing device 100 applies a corresponding transformation function TF to transform each stroke ST of text line LN1 into a structured handwritten format FT1. In this example, computing device 100 thus moves (translational movement) text line LN1 according to translation component CP1, re-scales text line LN1 according to scaling component CP2, and rotates text line LN1 according to rotation component CP3, as already described above with respect to Figure 11 The phrase “This example shows” once standardized as shown in Figure 11 still constitutes handwriting (formed by strokes), but is structured in a standardized manner.

[0199] In update step S22, computing device 100 also updates model data DT of text block BL1 based on the corresponding transformation function TF determined for text line LN1 in S18. More specifically, computing device 100 updates a part of model data DT associated with text line LN1 according to the corresponding transformation function TF calculated in S18.

[0200] In this example, during update step S22, computing device 100 updates line information 224 of model data DT associated with text line LN1 based on the corresponding transformation function TF determined in S18. Thus, the origin coordinates (x1, y1), inclination information (a1), and height information (h1) are updated according to components CP1, CP3, and CP2, respectively.

[0201] As indicated above, for each text line LN detected in line extraction step S10, steps S18, S20, and S22 are performed in a similar manner. Thus, text block BL1 undergoes normalization (S16), such that the text block is transformed (S20) into a structured format FT2, thereby resulting in the associated model data being updated (S22) accordingly. This normalization is performed line by line in the sense that each text line LN is normalized based on a dedicated transformation function TF and the model data associated with each text line LN is updated based on this dedicated transformation function TF.

[0202] Update step S22 can be performed before or after transformation step S20. More generally, a person skilled in the art can update the order of the sequence of performing steps S18 to S22 for different text lines LN.

[0203] As Figure 5 shown, the computing device can store (S24) the updated model data DT in memory 8 ( Figure 2 )). When performing update S22 for each text line LN of text block BL1, this storage step can be performed step by step. In this example, instead of storing the original model data DT in S14, the updated model data DT of text block BL1 is stored in S24.

[0204] In addition, the computing device 100 may display (S26) a normalized text block BL1, as Figure 10 depicted. Once normalized, the text block BL1 is arranged according to the structured format FT2 to comply with (match) the document pattern 200 as previously described.

[0205] The computing device 100 uses the origin and inclination defined in the corresponding line information 224 to draw the baseline 202 of each text line LN. Additionally, the height value of each text line LN may be represented on the display device 102 using a highlighted rectangle (or the like). When rescaling the text lines during normalization, the origin of each text line LN may be used as an anchor point.

[0206] Text normalization S16 of the text block BL1 from freehand format to a structured context or format (e.g., FT2) can improve the overall experience provided by ink interactivity. Normalization not only enables the arrangement and display of text handwritten input in a more uniform and structured manner, but also allows for an increase in the reliability and editing efficiency of the text recognition system, as explained below.

[0207] The present invention allows for the construction of digital ink with model data to form interactive ink. As previously indicated, the model data DT defines the mutual correspondence that links each stroke ST to a unique character CH, each character CH to a unique word WD, and each word WD to a unique text line LN, thereby enabling ink interactivity. For example, erasing a word in a text block means also erasing all the characters referenced by this word according to the document model. By way of another example, erasing all the content of a given line LN not only erases this line from the model data, but also causes the computing device 100 to erase all the constituent words WD and characters CH of this line. These correlations defined by the model data DT allow for the line-by-line normalization process as described above.

[0208] Due to the present invention, text handwritten input can be normalized while keeping the relevant model data representing the input text handwritten up-to-date, such that no additional text recognition needs to be performed in response to the transformations executed during the normalization process. In other words, as part of the normalization process, the computing device 100 updates the correlations between the strokes ST, characters CH, words WD, and text lines LN of the text handwritten input IN defined in the model data, such that no text recognition needs to be performed again on the normalized text block BL1, which might otherwise conflict with the results of the initial text recognition performed in S8.

[0209] Generally speaking, text recognition engines in known computing devices are typically configured to monitor any ink modifications or edits that occur on a text handwritten (e.g., deletion or editing of strokes). Thus, if text normalization occurs on such known computing devices, it will cause the text recognition engine to run again during normalization to attempt to recognize possible new content in the text handwritten input once it is converted into a structured format. In other words, normalization of digital ink in known systems results in discarding the previous recognition results and reprocessing the entire ink to obtain new recognition results. While such a responsive text recognition mechanism may be useful when, for example, some form of editing by the user in freehand mode causes ink modifications, the fact that computing device 100 performs new text recognition in response to the normalization S16 of text block BL1 from freehand format FT1 to structured format FT2 is actually counterproductive. Any additional text recognition in response to normalization S16 runs the risk of contaminating or conflicting with the initial text recognition S8 performed on the text handwritten input IN in freehand mode FT1. Additionally, performing such an additional text recognition process will require time and resources. Updating the model data DT according to the transformation function TF applied to the digital ink during normalization keeps the digital ink and associated data in a consistent and coherent state while saving time and resources.

[0210] Instead of triggering new text recognition in response to the normalization S16 performed on text block BL1, computing device 100 of the present invention is configured to update the model data DT of text block BL1 according to the transformation function TF applied to text line LN during normalization.

[0211] The gist of the normalization process in the present invention is to manipulate the digital ink for a given initial recognition result (i.e., the result of the initial text recognition S8). To this end, computing device 100 can transform the digital ink of text block BL1 into structured format FT2 and accordingly update the document model such that the result of the initial text recognition S8 obtained in freehand mode is not challenged.

[0212] In a particular embodiment, during normalization step S16, the model data DT of each text line LN is updated according to the corresponding transformation function TF obtained in S18 while preventing any text recognition caused by the application of the corresponding transformation function TF. Thus, computing device 100 prevents any new text recognition in response to the normalization performed on text block BL1 in step S16. Thus, even if the digital ink is transformed during normalization, the recognized text remains stable, thereby improving the reliability and efficiency of the entire recognition system.

[0213] However, it should be understood that once text block BL1 has been normalized to structured format FT2 (as Figure 10As shown in [reference], new text recognition can be triggered to reflect any subsequent form of editing or ink modification performed on the standardized text block BL1 at a later time. For example, if the user decides to edit the first text line LN1 in the structured mode FT2 (e.g., by erasing the strokes or parts of the strokes of the word WD or the character CH, or by correcting said strokes or parts of the strokes), the computer device 100 can perform new text recognition to identify text that may be different from the result of the initial text recognition S8.

[0214] Therefore, the computing device 100 preferably triggers new text recognition in response to an editing operation performed after the standardization step S16. In this case, the text recognition function is not permanently blocked in the computing device 100, but instead remains active after the completion of the standardization S16 to allow for the recognition of new content in the case of later editing of the text block BL1 in the structured format FT2. Thus, the computing device 100 can more reliably and easily process handwritten text for further operations and interactions, such as any form of editing (e.g., text reflow, etc.).

[0215] In addition, the present invention allows the user to input text handwriting in a freehand mode. Essentially, in the freehand mode, no constraints on lines, size, and orientation (including guide lines, margins, etc.) are imposed on the user, enabling the input of various complex forms of handwriting. By standardizing the text handwriting input into the structured handwriting format described previously, the computing device 100 can facilitate the implementation of editing functions (correction, rescaling, etc.) on the text handwriting. For example, the standardization can be generated by a user-initiated command received by the computing device or any predetermined event detected by the computing device. Even if the freehand mode was initially used, the user can edit and operate the handwritten input in a more structured and advanced manner, thereby enhancing the user experience.

[0216] For example, as Figure 12 shown, once the standardization S16 is completed, the computing device 100 can perform an editing operation during the editing step S28. In this example, text reflow is performed on the standardized text block BL1 in the horizontal direction (vertical text reflow is also possible). Thus, the relative positions of the different digital ink strokes of the text block BL1 in the structured format FT2 are rearranged. Such an editing operation can be triggered by the user in response to any suitable user command (e.g., a user gesture on the input surface 104), or by the computing device 100 itself after detecting a predetermined event.

[0217] Note that in some alternative embodiments, the functions recited in the blocks may not be performed in the order shown in the figures. For example, two blocks shown in succession may in fact be performed substantially concurrently, or the blocks may sometimes be performed in the reverse order or the blocks may be performed in an alternative order, depending upon the functionality involved.

[0218] The invention has been described in specific embodiments, and it is apparent that various modifications and embodiments can be provided within the capabilities of those skilled in the art, in accordance with the scope of the appended claims. Specifically, those skilled in the art can consider any and all combinations and variations of the various embodiments described in this document that fall within the scope of the appended claims.

Claims

1. A method for processing text handwriting implemented by a computing device (100), the computing device (100) including a processor, a memory, and at least one non-transitory computer-readable medium that processes text handwriting under the control of the processor, the method including: - Detect (S2) a plurality of input strokes (ST) of digital ink (IN) through an input surface (104), the input strokes being handwritten in a freehand format (FT1) without any handwriting constraints; - Display (S4) on a display device (102) the plurality of input strokes (ST) handwritten in the freehand format; - Classify (S6) each input stroke as text or non - text, the classification including detecting at least one text block (BL1) of handwritten text as text from the plurality of input strokes handwritten in the freehand format (FT1); - Perform (S8) text recognition on the at least one text block (BL1), the text recognition including: ○ Extract (S10) text lines of the handwritten text from the at least one text block; ○ Generate (S12) model data (DT) representing the handwritten text, the model data defining a correlation that associates each stroke (ST) of the at least one text block with characters (CH), words (WD), and text lines (LN) of the at least one text block; - Normalize (S16) each text line of the handwritten text from the freehand format to a structured format to comply with a document pattern (200), the document pattern including a line pattern, where the document pattern defines handwriting constraints that the handwritten text is to comply with, The normalization includes, for each text line of the handwritten text: ○ Calculate (S18) a corresponding transformation function (TF) for the text line to transform the text line (LN) into the structured format (FT2) such that the text line of the handwritten text is arranged according to a guiding line defined by the line pattern; ○ Apply (S20) the corresponding transformation function (TF) to the text line to transform each stroke of the text line into the structured format (FT2); ○ Prevent any text recognition of the at least one text block caused by applying the corresponding transformation function to the text line; and ○ Update (S22) the model data (DT) of the text line based on the corresponding transformation function, including updating the correlation between the strokes, characters, words, and text lines of the handwritten text in the model data.

2. The method according to claim 1, including storing the model data (DT) generated during the text recognition, wherein updating the model data further includes storing (S24) the updated model data of the at least one text block to replace the model data generated during the text recognition.

3. The method according to claim 1 or 2, including displaying (S26) the text lines of the at least one text block in the structured format (FT2) after the normalization.

4. The method according to claim 1 or 2, wherein the model data (DT) of the at least one text block (BL1) includes: - Define character information (220) for a plurality of characters, each character being associated with at least one stroke of digital ink and a text line of the at least one text block; - Define word information (222) for a plurality of words, each word being associated with at least one character defined by the character information; And - Define line information (224) for each text line of the at least one text block, each text line being associated with at least one word defined by the word information.

5. The method according to claim 4, wherein for each text line of the at least one text block, the line information includes: - Origin coordinates representing the origin of the text line; - Inclination information representing the inclination of the text line; And - Height information representing the height of the text line.

6. The method according to claim 4, wherein updating the model data (DT) during the normalization includes updating the line information (224) of the text line based on the corresponding transformation function.

7. The method according to claim 5, wherein the normalization comprises, for each text line: - determining input parameters, the input parameters including the origin coordinates, the tilt information, and the height information of the text line; wherein the corresponding transformation function is calculated based on the input parameters and the document pattern.

8. The method according to claim 1 or 2, wherein the document pattern defines at least one of the following handwriting constraints to be adhered to by the handwritten text: - the outer margin of the display area; and - the line spacing.

9. The method according to claim 1 or 2, wherein each transformation function defines at least one of the following transformation components to be applied to the corresponding text line during the normalization: - a translation component (CP1); - a scaling component (CP2); and - a rotation component (CP3).

10. The method according to claim 9, wherein the scaling component (CP2) of the transformation function is determined during the normalization based on the ratio of the distance between two consecutive guiding lines of the line pattern to the height of the corresponding text line.

11. The method according to claim 9, wherein the translation component of the transformation function is determined during the normalization to perform a translation of the text line such that the origin of the text line is moved to align with the corresponding guiding line of the line pattern, and the corresponding guiding line is assigned to the text line during the normalization.

12. The method according to claim 9, wherein the rotation component (CP3) is determined during the normalization to rotate the corresponding text line to reduce its tilt to zero according to the document pattern (200).

13. A computing device for handwritten text, comprising: - An input surface (104) for detecting a plurality of input strokes (ST) of digital ink (IN), the input strokes being handwritten in a freehand format (FT1) without any handwriting constraints; - A display device (102) for displaying the plurality of input strokes handwritten in the freehand format; - A classifier (MD2) for classifying each input stroke as text or non-text, the classifier being configured to detect at least one text block of handwritten text as text from the plurality of input strokes handwritten in the freehand format (FT1); - A line extractor (MD4) for extracting text lines of handwritten text from the at least one text block; - An identification engine (MD6) for performing text identification on each text line of the at least one text block, thereby generating model data representing the handwritten text, the model data defining a correlation associating each input stroke of the at least one text block with characters, words, and text lines of the at least one text block; - A text editor (MD8) for normalizing each text line of the handwritten text from the freehand format to a structured format to comply with a document mode, the document mode including a line mode, wherein the document mode defines handwriting constraints for the handwritten text to comply with, the text editor being configured to perform, for each text line of the handwritten text: ○ Calculating a corresponding transformation function for the text line to transform the text line into the structured format such that the text line of the handwritten text is arranged according to a guiding line defined by the line mode; ○ Applying the corresponding transformation function to the text line to transform each input stroke of the text line into the structured format; ○ Preventing any text identification of the at least one text block caused by applying the corresponding transformation function to the text line; and ○ Updating the model data of the text line based on the corresponding transformation function, including updating the correlation between the input strokes, characters, words, and text lines of the handwritten text in the model data.

Images