Reconstructing ink point sequences

Abstract

A system and method for reconstructing an ink point sequence from an image (IM) wherein the image depicts a captured handwriting input comprising: retrieving the image including pixels of the captured handwriting input; extracting a skeleton (SK) of pixels from the retrieved image; cutting the skeleton (SK) at junction pixels (1-14) and generating a plurality of segments (SG7-SG13); generating a plurality of sub-strokes as a plurality of sequences of coordinates of the segments wherein a segment provides two sub-strokes having opposite directions; generating a plurality of sub-stroke embeddings from the plurality of respective sub-strokes; using the plurality of sub-stroke embeddings to generate a set of probabilities for each sub-stroke of a sub-stroke sequence, each probability for a given sub-stroke corresponding to another sub-stroke; reconstructing the sequence of sub-strokes by selecting the sub-strokes with the highest probabilities; inferring the ink point sequence from the reconstructed sequence of sub-strokes.

Description

[0001] RECONSTRUCTING INK POINT SEQUENCES

[0002] Field of the invention

[0003] The present disclosure relates generally to the field of computing device interfaces capable of processing or recognizing user input handwriting of various graphics and text. In particular, the present disclosure concerns computing devices and corresponding methods for detecting and processing input handwritten elements.

[0004] Handwriting recognition can be implemented in computing devices to input and process various types of handwritten input elements, hand-drawn or handwritten by a user, such as text content (e.g., alphanumeric characters) or non-text content (e.g., shapes, drawings). Once inputted on a computing device, the input elements are usually displayed as digital ink and undergo handwriting recognition to be manipulated as objects, for example text objects such as characters, words, lines or paragraphs or non-text objects such as primitives or shapes. The user handwriting input is typically interpreted using a real-time handwriting recognition system or method.

[0005] Strokes entered on or via the input surface are processed by the processor as digital ink. A user may enter a stroke with a finger or some instrument such as a pen or stylus suitable for use with the input surface. The user may also enter a stroke by making a gesture above the input surface if technology that senses motions in the vicinity of the input surface is being used, or with a peripheral device of the computing device, such as a mouse or joystick. A stroke is characterized by at least the stroke initiation location, the stroke termination location, and the path connecting the stroke initiation and termination locations. Because different users may naturally write the same object, e.g., a letter, a shape, a symbol, with slight variations, the HWR system accommodates a variety of ways in which each object may be entered whilst being recognized as the correct or intended object.

[0006] Note-taking which is a ground to express our volatile thoughts and ideas, is still a manual procedure. However, while pen and paper still offer unmatched comfort and efficient input methods for handwritten notes, it disables their exploitation to their full potential. Other techniques of handwriting recognition may be applied, such as offline recognition, referring to techniques of recognition of digital ink as static images usually digitized as offline documents by capturing images with a scanner or camera. This is an inconvenient step for most users and it also adds noise that affects offline processing systems.

[0007] Handwritten text content of imaged input, so-called off-line handwriting recognition, is conventionally recognized using intelligent character recognition (ICR) or intelligent word recognition (IWR). ICR systems recognize content from images of characters handwritten in a structured manner. This structure includes characters which are isolated, e.g., non-cursive, and substantially evenly-spaced. ICR has been successful in enterprise form applications in which users handwrite information in fields or boxes designated for individual characters, such as letters, numbers and symbols. IWR systems on the other hand recognize content from images of characters handwritten in an unstructured or free-form manner, e.g., cursive. This is achieved by performing word-based recognition, rather than character-based recognition as in ICR.

[0008] Like OCR (Optical Character Recognition), ICR and IWR conventionally involve image processing techniques for resolving text characters from the image, including skeletonization. Also like OCR, the accuracy of the recognition using either ICR or IWR depends on the quality of the imaged characters. This is exacerbated for handwriting however as handwriting includes a wide range of natural variation, many differing styles and may not adhere to expected standards. As such, the technique of thinning pixels in images of handwritten content may introduce unwanted trajectories in the handwriting strokes, such as branches, segments, tails, “neckings”.

[0009] Thus, the conventional offline handwriting recognition techniques such as OCR, ICR, IWR, etc. are usually limited in terms of efficiency, often resulting in limited recognition accuracy and reliability.

[0010] Summary of the invention

[0011] For handwriting applications, the availability of temporal information in online systems often makes them better performing than their offline analog. Online handwriting recognition systems are usually more efficient than offline systems because they capture dynamic temporal information, such as the sequence, direction, and speed of writing, which enhances accuracy. This real-time data allows for immediate processing and feedback, making online recognition more responsive and effective for applications like digital note-taking for instance. In contrast, offline handwriting recognition is based on analyzing static images of the handwritten content, thus lacking the detailed stroke information that may be used to distinguish similar characters and improve recognition accuracy.

[0012] Therefore, an aim of the present invention is to permit recovery of pen trajectory from images, that is, recovering temporal information of ink points comprised in handwriting, thereby allowing the users to take to their advantage the best of the two modalities: ergonomic note-taking with a pen and paper and powerful editing and processing of the digital ink.

[0013] The examples of the present invention that are described herein below provide computing devices, methods and corresponding computer programs for reconstructing an ink point sequence from an image wherein the image depicts a captured handwriting input.

[0014] According to a first aspect, the invention provides a method, implemented by a computing device as defined below, for ink point sequence reconstruction, said method comprising:

[0015] - retrieving an image depicting a captured handwriting input (IN), said image including pixels of the captured handwriting input;

[0016] - extracting a skeleton of pixels from the retrieved image;

[0017] - cutting the skeleton at junction pixels into a plurality of segments;

[0018] - generating a plurality of sub-strokes based on said segments, each sub-stroke comprising a sequence of coordinates of a respective one of said segments, wherein each segment comprises a first end and a second end and each segment comprises two sub-strokes having opposite directions, the first substroke starting at the first end and ending at the second end and the second sub-stroke starting at the second end and ending at the first end;

[0019] - generating a plurality of sub-stroke embeddings based respectively on the plurality of sub-strokes;

[0020] - generating, based on the plurality of sub-stroke embeddings, a set of probabilities for each sub-stroke of the plurality of sub-strokes, each probability for a given sub-stroke defines the likelihood that another sub-stroke being the subsequent sub-stroke to the given sub-stroke in a sequence of sub-strokes;

[0021] - reconstructing the sequence of sub-strokes by selecting the sub-strokes with the highest probabilities; and

[0022] - inferring a temporal sequence of ink points of the captured handwritten input based on the reconstructed sequence of sub-strokes.

[0023] According to a particular embodiment, the generating of a set of probabilities for each sub-stroke of the plurality of sub-strokes comprises: setting a given sub-stroke embedding as an initial embedding; generating a set of probabilities of a following substroke by applying the given sub-stroke embedding to an ordering-transformer; determining the following sub-stroke as the highest probability of the set of probabilities of the given sub-stroke; concatenating the determined following sub-stroke to the end of the sequence of sub-strokes; setting the given sub-stroke embedding as the determined following sub-stroke respective embedding; repeating, for each given substroke embedding, the generating, the determining, the concatenating, the setting until the given sub-stroke embedding corresponds to the last sub-stroke of the sequence.

[0024] According to a particular embodiment, one of the probabilities of the set of the probabilities corresponds to the end of the sequence, following the given sub-stroke.

[0025] According to a particular embodiment, the extracting of the skeleton is performed by applying the image to a fully convolutional neural network (FCNN).

[0026] According to a particular embodiment, the cutting of the skeleton is performed by removing junction pixels wherein a junction pixel is a pixel from the skeleton with 3 or more 8-connected skeleton pixel neighbor.

[0027] According to a particular embodiment, the generating of a plurality of sub-stroke embeddings from the plurality of respective sub-strokes comprises: first, projecting the respective sequences of coordinates of the plurality of sub-strokes within a plurality of respective sub-stroke vectors; then, transforming the plurality of sub-stoke vectors into the plurality of respective sub-stroke embeddings of by applying the plurality of substroke vectors to an embedding-transformer.

[0028] According to a particular embodiment, the plurality of sub-stroke vectors have a fixed first-size. According to a particular embodiment, the plurality of sub-strokes embeddings have a fixed second-size.

[0029] According to a particular embodiment, the plurality of sub-stroke embeddings are reconstructed as a plurality of respective parametric approximation of sub-stroke curves by an embedding-decoder, wherein the embedding-transformer is trained according to the reconstructed sub-strokes.

[0030] According to a particular embodiment, the generating a set of probabilities of a following sub-stroke, comprises, for each given sub-stroke applied to the orderingtransformer predicting a pen state.

[0031] According to a particular embodiment, the inferring of the ink point sequence from the reconstructed sequence of sub-strokes includes interpolating ink points between two consecutive sub-strokes when the pen state of the earlier sub-stroke is pen-down.

[0032] Accordingly, the invention according to a second aspect also provides a computer program that may be stored on a recording medium, this computer program being arranged to be implemented by one or more processors or by a computer, this computer program comprising instructions adapted for the implementation of a method according to the first aspect of the invention.

[0033] The computer programs of the invention can be expressed in any programming language, and can be in the form of source code, object code, or any intermediary code between source code and object code, such that in a partially-compiled form, for instance, or in any other appropriate form.

[0034] The invention according to a third aspect also provides a recording medium readable by one or more processors or by a computer, this recording medium comprising computer program instructions as mentioned above.

[0035] The recording medium previously mentioned can be any entity or device capable of storing the computer program. For example, the recording medium can comprise a storing means, such as a ROM memory (a CD-ROM or a ROM implemented in a microelectronic circuit), or a magnetic storing means such as a floppy disk or a hard disk for instance.

[0036] The recording medium of the invention can correspond to a transmittable medium, such as an electrical or an optical signal, which can be conveyed via an electric or an optic cable, or by radio or any other appropriate means. The computer program according to the invention can in particular be downloaded from the Internet or a network of the like.

[0037] Alternatively, the recording medium can correspond to an integrated circuit in which a computer program is loaded, the circuit being adapted to execute or to be used in the execution of the methods of the invention.

[0038] According to a fourth aspect, the present invention also relates to a corresponding computing device, for ink point sequence reconstruction, configured to implement the method of the first aspect of the invention.

[0039] In a particular embodiment, the computing device comprises:

[0040] - an image retrieving module configured to retrieve an image depicting a captured handwriting input, said image including pixels of the captured handwriting input;

[0041] - a skeleton extracting module configured to extract a skeleton of pixels from the image;

[0042] - a skeleton cutting module configured to cut the skeleton at junction pixels into a plurality of segments;

[0043] - a sub-stroke generating module configured to generate a plurality of sub-strokes based on said segments, each sub-stroke comprising a sequences of coordinates of a respective one of said segments, wherein each segment comprises a first end and a second end and each segment comprises two substrokes having opposite directions, the first sub-stroke starting at the first end and ending at the second end and the second sub-stroke starting at the second end and ending at the first end;

[0044] - an embedding generating module configured to generate a plurality of substroke embeddings based respectively on the plurality of sub-strokes;

[0045] - a sub-stroke sequence reconstructing module configured to: o generate, based on the plurality of sub-stroke embeddings, a set of probabilities for each sub-stroke of the plurality of sub-strokes, each probability for a given sub-stroke defines the likelihood that another substroke being the subsequent sub-stroke to the given sub-stroke in a sequence of sub-strokes; and o reconstruct the sequence of sub-strokes by selecting the sub-strokes with the highest probabilities;

[0046] - an ink point sequence inferring module configured to infer a temporal sequence of ink points of the captured handwritten input based on the reconstructed sequence of sub-strokes.

[0047] According to a particular embodiment, the sub-stroke sequence reconstructing module comprises: o a sub-stroke vectorizer configured to linearly project a sequence of ink points of a sub-stroke into a sub-stroke vector; and o an embedding-transformer, coupled to the sub-stroke vectorizer module, configured to generate a sub-stroke embedding from the respective substroke vector.

[0048] According to a particular embodiment, the sub-stroke sequence reconstructing module comprises: o an ordering-transformer configured to generate, for a sub-stroke, a probability of a pen state and extract a set of probabilities of a following substroke; o a subsequent sub-stroke determining module, coupled to the ordering transformer, configured to determine the following sub-stroke of a substroke as the highest probability of the set of probabilities; and o a sub-stroke concatenating module, coupled to the subsequent sub-stroke determining module, configured to concatenate the determined following sub-stroke to the end of the sequence of sub-strokes.

[0049] According to a particular embodiment, the computing device further comprising a transformer decoder configured to reconstruct the plurality of sub-stroke as a plurality of respective parametric approximation of the sub-stroke.

[0050] The various embodiments defined above in connection with the method according to the first aspect of the present invention apply in an analogous manner to the computing device of the second aspect, as well as the computer program and the non-transitory computer readable medium of the present disclosure. For each step of the method of the present invention as defined in the present disclosure, the computing device may comprise a corresponding module configured to perform said step, and vice versa.

[0051] The present invention advantageously allows recovering in an efficient and reliable manner temporal information related ink points from an image of handwritten content. This temporal information defines how the handwritten content, and more precisely ink points forming such handwritten content, have been inputted over time (information on pen trajectory over time), for instance by a user using any suitable input technique. By inferring a temporal sequence of ink points forming the handwritten content, said content can then be efficiently processed based on this information, for instance to achieve improved handwriting recognition.

[0052] Brief description of the drawings

[0053] The present system and method will be more fully understood from the following detailed description of the examples thereof, taken together with the drawings. In the drawings like reference numerals depict like elements. In the drawings:

[0054] FIG. 1 shows a block diagram of a computing device in accordance with an embodiment of the present invention;

[0055] FIG. 2 shows a flow diagram of an example of the present method in accordance with an embodiment of the present invention;

[0056] FIG. 3 shows a block diagram of a system for an ink point sequence reconstruction in accordance with an embodiment of the present invention;

[0057] FIG. 4A and 4B show a flow diagram illustrating an example of ink point sequence reconstruction of FIG. 3.

[0058] Detailed description

[0059] In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without such details. In other instances, well known methods, procedures, and / or components are described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0060] The following description of the exemplary embodiments refers to the accompanying drawings. The following detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims. In various embodiments as illustrated in the figures, a computing device, a corresponding method and a corresponding computer program are discussed.

[0061] The use of the term "text" in the present description is understood as encompassing all characters (e.g. alphanumeric characters or the like), and strings thereof, in any written language and any symbols used in written text.

[0062] The term "non-text" in the present description is understood as encompassing freeform handwritten or hand-drawn content (e.g. shapes, drawings, etc.) and image data. Nontext content defines graphic or geometric formations in linear or non-linear configurations, including containers, drawings, common shapes (e.g. arrows, blocks, etc.) or the like.

[0063] The terms "hand-drawing" and "handwriting" are used interchangeably herein to define the creating of digital contents by users through use of their hands (or fingers) or an input device (hand-held stylus or digital pen, mouse...) on or with an input surface 4. The term "hand" or the like is used herein to provide concise description of the input techniques, however the use of other parts of a user's body for similar input is included in this definition, such as foot, mouth and eye.

[0064] A handwritten content in the present disclosure is understood as comprising one or multiple symbols (or characters), or a part thereof, created by a user through a process of handwriting in any suitable manner. Handwriting input may for instance comprise text and / or non-text content. Text may comprise various contents, such as characters and the like (e.g. alphanumeric characters), or strings thereof, in any written language and, more generally, any symbol, or part thereof, used in written text. Non-text content may comprise various contents, such as any handwritten shape, drawing, etc.

[0065] The various technologies described herein generally relate to capture, processing and management of hand-drawn or handwritten content on portable and non-portable computing devices, referenced hereinafter as DV, depicted in figure 1. The systems and methods described herein may utilize recognition of user's natural writing and drawing styles input to a computing device via an input surface 4, such as a touch sensitive screen (as discussed later).

[0066] In the following, it is assumed that the computing device DV is used to handwriting input by implementing a method for point sequence reconstruction according to particular embodiments of the invention. Some implementation details are provided herebelow for purely illustrative purpose, other embodiments being possible according to the present disclosure.

[0067] The computing device (or digital device) DV may be a computer desktop, laptop computer, tablet computer, e-book reader, mobile phone, smartphone, wearable computer, digital watch, interactive whiteboard, global positioning system (GPS) unit, enterprise digital assistant (EDA), personal digital assistant (PDA), game console, or the like. The computing device DV includes components of at least one processing elements, some form of memory and input and output (I / O) devices. The components communicate with each other through inputs and outputs, such as connectors, lines, buses, links networks, or others known to the skilled person.

[0068] As shown in figure 1 , the computing device DV may comprise a user interface 4 comprising at least one display 5 for outputting data from the computing device such as images, text, and video. The display 5 may use LCD, plasma, LED, OLED, CRT, or any other appropriate technology that is or is not touch sensitive as known to those of ordinary skill in the art. At least part of the display is co-located with at least one input area. The input area may employ technology such as resistive, surface acoustic wave, capacitive, infrared grid, infrared acrylic projection, optical imaging, dispersive signal technology, acoustic pulse recognition, or any other appropriate technology as known to those of ordinary skill in the art to receive user input. The input area may be bounded by a permanent or video-generated border that clearly identifies its boundaries. Instead of, or additional to, an on-board display, the computing device DV may have a projected display capability.

[0069] The user interface 4 of the computing device DV may comprise an input interface 4 (figure 1 ) also other embodiments without such an input interface are also possible.

[0070] The computing device DV also includes a processor 6, which is a hardware device for executing software, particularly software stored in a memory 7 (figure 1 ). The processor 6 can be any custom made or commercially available general purpose processor, a central processing unit (CPU), commercially available microprocessors including a semiconductor based microprocessor (in the form of a microchip or chipset), microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, state machine, or any combination thereof designed for executing software instructions known to those of ordinary skill in the art.

[0071] The memory 7 can include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, or SDRAM)) and nonvolatile memory elements (e.g., ROM, EPROM, flash PROM, EEPROM, hard drive, magnetic or optical tape, memory registers, CD-ROM, WORM, DVD, redundant array of inexpensive disks (RAID), another direct access storage device (DASD), or any other magnetic, resistive or phase-change nonvolatile memory). Moreover, the memory 7 may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory can have a distributed architecture where various components are situated remote from one another but can also be accessed by the processor. Further, the memory 7 may be remote from the device, such as at a server or cloud-based system, which is remotely accessible by the computing device DV. The memory 7 is coupled to the processor 6, so the processor can read information from and write information to the memory 7. In the alternative, the memory 7 may be integral to the processor 6. In another example, the processor 6 and the memory 7 may both reside in a single ASIC or other integrated circuit.

[0072] The software in the memory 7 may include an operating system 8 and an application 12 in the form of a non- transitory computer readable medium having a computer readable program code embodied therein. The operating system 8 controls the execution of the application 12. The operating system 8 may be any proprietary operating system or a commercially or freely available operating system, such as WEBOS, WINDOWS®, MAC and IPHONE OS®, LINUX, and ANDROID. It is understood that other operating systems may also be utilized. Alternatively, the application of the present system and method may be provided without use of an operating system. The application 12 (or computer program) includes one or more processing elements related to detection, management and treatment of user input (discussed in detail later). This computer program 12 may comprise instructions to implement a method according to particular embodiments of the invention, as further described below. The memory 7 may thus constitute a recording medium (or information medium) according to particular embodiments of the present invention, having stored thereon the computer program 12.

[0073] The software stored in memory 7 may also include one or more other applications related to handwriting recognition (HWR), different functions, or both. Some examples of other applications include a text editor, telephone dialer, contacts directory, instant messaging facility, computer-aided design (CAD) program, email program, word processing program, web browser, and camera.

[0074] The application 12, with support and compliance capabilities, may be a source program, executable program (object code), script, application, or any other entity having a set of instructions to be performed. When a source program, the program needs to be translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory, so as to operate properly in connection with the operating system. Furthermore, the HWR system with support and compliance capabilities can be written as (a) an object oriented programming language, which has classes of data and methods; (b) a procedure programming language, which has routines, subroutines, and / or functions, for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol, Perl, Java, Objective C, Swift, and Ada; or (c) functional programing languages for example but no limited to Hope, Rex, Common Lisp, Scheme, Clojure, Racket, Erlang, OCaml, Haskell, Prolog, and F#.

[0075] Strokes may be entered or obtained by the computing device DV in different manners, for instance on or via the input surface 4 (for instance a touch based user input interface) are processed by the processor 6 as digital ink. A user may enter a stroke with a finger or some instrument such as a pen or stylus suitable for use with the input surface 4. The user may also enter a stroke by making a gesture above the input surface 4 if technology that senses motions in the vicinity of the input surface 4 is being used, or with a peripheral device of the computing device, such as a mouse or joystick. In the present disclosure, a stroke (or ink stroke) is characterized by at least the stroke initiation location (corresponding to a “pen down” event), the stroke termination location (corresponding to a “pen up” event), and the path connecting the stroke initiation and termination locations. Because different users may naturally write the same object, e.g., a letter, a shape, a symbol, with slight variations, the HWR system accommodates a variety of ways in which each object may be entered whilst being recognized as the correct or intended object.

[0076] In accordance with a particular embodiment of the present invention, a method for reconstructing an ink point sequence from an image (also called hereinafter method for ink point sequence reconstruction), which is implemented by the computing device DV, is now described with reference to Figure 2. This method involves processing an image depicting a captured handwriting input to infer therefrom temporal information relative to the handwriting input. More specifically, the computing device DV as previously described (figure 1 ) implements this method by executing the application 12 stored in the memory 7.

[0077] In a retrieving step S110, the computing device DV retrieves the image including pixels of the captured handwriting input. This image depicts the captured handwriting input which is to be processed by the present method. The image may be obtained by the computing device DV in any suitable manner.

[0078] The image may include a pixel representation of digitized handwritten content, which may be of various kind depending on the case.

[0079] In a particular example, the image comprises a digital document containing the handwritten content. This document may for instance be produced by an imaging device, such as camera, that can be used to image or scan content from non-digital sources, such as a user handwritten input onto paper notes or white boards.

[0080] This captured content may be saved as an image document on the capturing device and communicated with another device, for example a smartphone or a scanner communicating with the computing device.

[0081] In an extracting step S120, the computing device DV extracts a skeleton of pixels from the retrieved image. This pixel skeleton represents for instance a 1 -pixel thick outline of the captured handwritten input. An example of such a pixel skeleton is depicted in figure 4A for illustrative purpose only.

[0082] In a particular example, the skeleton of the imaged handwriting input is a compact and non-redundant trace of 1 -pixel thick outline of the captured handwritten input. The image processing techniques for resolving (or determining) the handwriting input trace from the image typically involve analysis of the pixels of the digitized image in ‘monochromatic’, grayscale or color domains to extract pixels which can be considered to represent the handwritten trace that can be considered to represent the user input.

[0083] In one example, extracting the skeleton of the imaged handwriting input may involve thinning of foreground pixels in the image in iterative, single-pass or parallel processing fashion. This method may be performed by removing foreground pixels while thinning the objects.

[0084] In another example, gradient-based methods involve computing gradients which are highest at boundary pixels of objects and lowest at the skeleton pixels. These gradients can be calculated using distance transforms on the binary or grayscale image intensities. Contour-based methods, such as medial axis transform, techniques with Delaunay triangulation and Voronoi diagrams, use the perimeter of the object to deduce the skeleton. Principal component analysis (PCA) methods have also been explored in which the skeleton is approximated as a set of parametric curves, like Bezier or B-splines, and fitted to the image data.

[0085] Yet, in another example, extracting the skeleton of the imaged handwriting input may involve using a fully convolutional neural network (FCNN) consisting of a downsampling and an upsampling process as further explained below and in the publication. Applying End-to-End Trainable Approach on Stroke Extraction in Handwritten Math Expressions Images. In Mohamed Moussa, E., Lelore, T., & Mouchere, H. (2021 ) Document Analysis and Recognition-ICDAR 2021 : 16th International Conference, Lausanne, Switzerland, September 5-10, 2021 , Proceedings, Part III 16 (pp. 445-458). Springer International Publishing.

[0086] In a particular embodiment, an additional skeleton thinning step S122 may be performed as part of the extracting step S120 to remove remaining ambiguities along the skeleton path, thereby thinning down the skeleton to unitary thickness as further explained in the publication of Zhang, T. Y., & Suen, C. Y. (1984). A fast parallel algorithm for thinning digital patterns. Communications of the ACM, 27(3), 236-239.

[0087] In a skeleton cutting step S132 (figure 2), the computing device DV cuts (or divides) the skeleton into a plurality of segments. This cutting is performed at junction pixels of the skeleton. A junction pixel is defined as a skeleton pixel with three or more 8- connected skeleton pixel neighbors, defining an intersection or overlapping paths of the pixel skeleton. On a two-dimensional square lattice, a 8-connected pixel is composed of a central cell and the eight cells that surround it. Skeleton pixels with exactly one 8-connected skeleton neighbor define one extremity of a segment, wherein the segment trace has two extremities.

[0088] In particular, the computing device DV may remove different junction pixel S132 from the pixel skeleton, this corresponding to removing a pixel as foreground pixel and setting it as background pixel. This cutting is performed at junction pixels of the skeleton by removing the junction pixels and defining the resulting connected skeleton pixels as segments.

[0089] Such a cutting step S132 may be performed for any handwritten content and results in a simplified definition of the 1 -pixel thick outline of the segment. Indeed, an intersection of the handwritten input trace, captured as an image, may be represented by at least one one-pixel intersection. Such one-pixel intersection carries data of at least two segments having at least two original directions. For further processing the reconstruction of the ink point sequence, the cutting step advantageously removes potential ambiguities of the one-pixel intersection to only keep single-direction pixels to form the segments.

[0090] In a particular example, a fully convolutional neural network (FCNN) is used to extract a skeleton (e.g. 1 -pixel thick outline) from the image. A sub-stroke cutting algorithm may then be used based on junction detection within pixels of the segments.

[0091] In a sub-stroke generating step S134, the computing device DV generates a plurality of sub-strokes based on the segments obtained in cutting step S132. Each sub-stroke comprises a sequence of pixel coordinates of a respective one of said segments.

[0092] In the generating step S134, each segment may be defined as comprising two substrokes having opposite directions (or pixel orders), the first sub-stroke starting at the first end and ending at the second end and the second sub-stroke starting at the second end and ending at the first end. As shown in Figure 4B, for example, the segment SG10 comprises two respective sub-strokes SS101 and SS102, wherein

[0093] 55101 is defined as a first sequence from point P1 to point P27 (first direction) and

[0094] 55102 is a second sequence from point P27 to point P1 (second direction opposite to the first direction). In other words, each sub-stroke contained in a segment may comprise a first sequence of coordinates of ink points ordered from a first end to a second end of the sub-stroke and a second sequence of coordinates of the ink points ordered from the second end to the first end, i.e. according to an order opposite (inverse) to the order of the first sequence. This means that, given a set of sub-strokes, with a sub-stroke defined as a sequence of pixel coordinates, each (or a least one) sub-stroke from the skeleton may appears twice, in both directions. This advantageously allows predicting the sequence indicating the writing order of the different sub-strokes and how they should be merged to form strokes.

[0095] In a particular example, the computing device DV computes (S134) the plurality of substrokes S134 by defining a plurality of sub-strokes including a plurality of first substrokes as the paths from one extremity to another respective extremity of each segment and a plurality of second sub-strokes as respective opposite traversal paths of each segment. The two opposite paths of each segment are considered as possible hypotheses of the original user’s trace of the handwriting input. Since the user trace is not known, the reconstruction of an ink point sequence corresponding to the respective segment takes into account traces starting from either end.

[0096] In a sub-stroke embedding generating step S140, the computing device DV generates a plurality of sub-stroke embeddings based respectively on the plurality of sub-strokes obtained in the generating step S134. Each sub-stroke embedding corresponds to high-level annotations of sub-stroke features such as the shape, size and / or curvature of the handwriting input. These sub-stroke embeddings may be computed by a neural network transformer.

[0097] In particular example, the computing device DV generates (S140) a plurality of substroke vectors along with the embeddings based on the plurality of sub-strokes respectively. The plurality of sub-strokes may be first projected into a fixed dimension space as a plurality of respective sub-stroke vectors and further transformed as a plurality of respective sub-stroke embeddings. In a particular example, the generating step S140 comprises a sub-stroke vectorizing step S142 and sub-stroke transforming step S144, shown as module 142 and 144 respectively on figure 3 and as further described below.

[0098] First, in the sub-stroke vectorizing step S142, the computing device DV may outline each sub-stroke as a sequence of time points, wherein the time points of each substroke are linearly projected within a respective sub-stroke vector of a fixed first size, for example the fixed first size of the sub-stroke first vector is 64. Therefore, the sub strokes points are projected into a fixed dimensional latent space represented by a vector.

[0099] Additionally, the sub-stroke points may be shifted to start at an origin of the image and normalized relative to the offline image dimensions. This advantageously ensures that the vectors only capture important local geometric features by preventing the model from accessing any global information such as the canvas position or existence of other strokes and their inter-dependencies.

[0100] In one example, a sinusoidal positional encoding is added to each sub-stroke points vectors. The sinusoidal positional encoding advantageously assigns to each substroke points a unique representation.

[0101] Then, in the sub-stroke transforming step S144, the computing device DV generates a plurality of sub-stroke embeddings by applying the plurality of respective vectors to an embedding-transformer. The embedding-transformer may be transformer neural network by tracking relationships in sequential data such as the sequence of vectors corresponding to the sub-stroke points. The embedding-transformer generates an abstraction of the variable-length sub-strokes, defined in the cutting step S132, by encoding them into fixed-length embeddings. The embedding-transformer captures local information such as the shape, size, or curvature of the handwriting input into features of high-level annotations of the sequences of ink points of the sub-strokes.

[0102] In a particular embodiment, the plurality of the embeddings may be recomputed as a plurality of respective sub-strokes. The sub-strokes are reconstructed as a parametric approximation of a sub-stroke curve, such as coordinates (X, Y) of the sub-stroke curve at every timestep, generated by an embedding-decoder. The sub-stroke embeddings are input to the embedding-decoder. The embedding-transformer may be trained according to the reconstructed sub-strokes. The embedding-decoder may be a two-layer multilayer perceptron comprising, for example, a hidden layer and an output layer of two nodes wherein each node generates each coordinate (X and Y) of an inferred sequence of ink points. The embedding transformer including the embedding decoder may be trained with any loss function calculated based on the coordinates of a set of sub-strokes and coordinates of the reconstructed sub-strokes (i.e. based on the difference between the resulting sequence of ink points and the training data).

[0103] In a generating step S150, using the plurality of sub-stroke embeddings, the computing device DV generates a set of probabilities for each sub-stroke of the plurality of substrokes to be the subsequent sub-stroke in a sub-stroke sequence, whereby each probability for a given sub-stroke defines a likelihood that another sub-stroke is the subsequent (or next, or following) sub-stroke to the given sub-stroke in the sub-stroke sequence. This sub-stroke sequence is a temporal sequence of the generated substrokes that is progressively constructed (or generated) by the computing device DV according to the present method, as explained further below. Each subsequent substroke of the sub-stroke sequence comprises the sequence of coordinates of ink points for reconstructing the ink point sequence of the captured handwriting input.

[0104] By the term “subsequent” sub-stroke, it is meant the sub-stroke directing succeeding the given sub-stroke according to an order of a temporal sequence of sub-strokes.

[0105] In a particular embodiment, a given sub-stroke embedding is set as an initial embedding, therefore the computing device DV generates, in the generating step S150, the set of probabilities of a subsequent (or following) sub-stroke by applying the given sub-stroke embedding to an ordering-transformer.

[0106] In a determining step S160, the subsequent sub-stroke is determined as the sub-stroke with the highest probability of the set of probabilities of the given sub-stroke. In another word, the sub-stroke with the highest probabilities is selected (S160) as the sub-stroke subsequent to the given sub-stroke. The sequence of sub-strokes is thus reconstructed by adding (or inserting) therein the subsequent sub-stroke.

[0107] In a particular example, in a concatenating (or reconstructing) step S170, the computing device DV concatenates the subsequent sub-stroke determined in step S160 to the end of the sequence of sub-strokes. The given sub-stroke embedding (i.e. the embedding of the given sub-stroke) is therefore set as the determined subsequent sub-stroke respective embedding (i.e. the embedding of the subsequent sub-stroke) and for each given sub-stroke embedding the steps of generating S150, determining S160, concatenating S170 and setting each given sub-stroke embedding is repeated until the given sub-stroke embedding corresponds to the last sub-stroke of the sequence.

[0108] In other words, the computing device DV may perform iteratively the following cycle (or iterative process):

[0109] - generating (S150) a set of probabilities of sub-strokes (other than the given substroke) being the subsequent sub-stroke of the given sub-stroke by applying the given sub-stroke embedding to an ordering-transformer;

[0110] - determining (S160), as the subsequent sub-stroke of the given sub-stroke, the sub-stroke with the highest probability of the set of probabilities;

[0111] - concatenating (S170) the determined subsequent sub-stroke to the end of the sequence of sub-strokes;

[0112] - setting the given sub-stroke’s embedding as the embedding of the determined subsequent sub-stroke; wherein said cycle is repeated iteratively, for each given sub-stroke embedding, until the given sub-stroke embedding corresponds to the last sub-stroke of the sequence.

[0113] A given sub-stroke embedding may be set as an initial embedding when performing the above iterative cycle for the first time (first iterative round). The given sub-stroke embeddings being processed in the following rounds of the iterative cycle are defined as part of the iterative cycle (see above the setting step).

[0114] In a particular embodiment, in the generating step S150, the computing device DV extracts a set of probabilities that each of the plurality of sub-strokes (obtained in the generating step 134) is a subsequent (or following) sub-stroke of a current (or given) sub-stroke by applying a current (or given) sub-stroke embedding to an ordering transformer.

[0115] An ordering transformer may be trained, as further explained below, to predict a probability of a current pen state indicating if a current sub-stroke is the end of a user original handwritten stroke, referred to as pen-up state, or if the current sub-stroke is, in the user original handwritten stroke, concatenated with the following sub-stroke, referred to as pen-down state.

[0116] From the ordering transformer processing, the computing device DV may also extract a current set of probabilities that each of the plurality of sub-strokes is the following (or subsequent) sub-stroke of the current sub-stroke.

[0117] At first, in an initial stage, the computing device DV may apply to the ordering transformer an initial sub-stroke embedding representing a theoretical beginning of a sub-stroke sequence as the current sub-stroke.

[0118] Therefore, during the initial stage, the computing device DV may extract, from the ordering transformer, the initial set of probabilities that each of the plurality of substrokes is the following sub-stroke of the beginning of the sequence.

[0119] In response to the step S150, in a subsequent (or following) sub-stroke determining step S160, the computing device DV determines the subsequent (or following) substroke of the current sub-stroke as the highest probability of the current set of probabilities.

[0120] The subsequent (or following) sub-stroke becomes the current sub-stroke and in a subsequent iteration of the next sub-stroke probabilities calculating step S150, the computing device DV applies the respective sub-stroke embedding to the orderingtransformer.

[0121] In a particular embodiment, the respective sub-stroke embedding applied to the ordering-transformer is concatenated with sub-stroke data such as a positional embedding of a spatial arrangement of the respective sub-stroke in the image, for example the positional embedding is (or comprises) coordinates of the starting point of the sub-stroke.

[0122] Accordingly, the computing device DV may compute, iteratively, the set of probabilities calculating step S150 and the following sub-stroke determining step S160.

[0123] The step S150 allows to predict the updated probability of the current pen state and extract the updated current set of probabilities that each of the plurality of sub-strokes is the following sub-stroke of the current sub-stroke. Then in response to S150, the step S160 allows to determine the highest probability of the current set of probability as the subsequent (or following) sub-stroke and to set the following sub-stroke as the updated current sub-stroke.

[0124] Finally, in a final stage of the step S160, the computing device DV may determine that the highest probability of the current set of probability is a final state, representing a theoretical end of the sub-stroke sequence as the following sub-stroke and to stops any subsequent iteration of S150.

[0125] Still in reference to figure 2, in a sub-stroke sequence reconstructing step S170, the computing device DV reconstructs the sequence of sub-strokes by concatenating each determined subsequent (or following) sub-stroke of each iteration of step S160 from the initial state until the determined final state, resulting in a predicted sequence of sub-strokes.

[0126] In an ink point sequence inferring step S180, the computing device DV infers the sequence of ink points of the captured handwriting input from the reconstructed sequence of sub-strokes. The sequence of ink points may be linearly interpolated to fill the void left between two consecutive sub-stroke extremities when the pen state of the earlier sub-stroke is predicted as a pen down.

[0127] According to a particular embodiment, when running the applications 12 stored in the memory 7 (figure 1 ), for each step of the method of the present invention, the computing device DV may comprise a corresponding module configured to perform said step, namely: an image retrieving module 110, a skeleton extracting module 120, a skeleton cutting module 132, a sub-stroke generating module 134, a sub-stroke embedding generating module 140, a subsequent sub-stroke probabilities calculating module 152, a subsequent sub-stroke determining module 154, a sequence of substroke reconstructing module (also called sub-stroke concatenating module) 156 and a ink point sequence inferring module (also called ink point inferring module) 160.

[0128] The above description of each step of the method represented in figure 2 may apply in an analogous manner to each corresponding module of the computing device DV as shown in figure 1 and further described in figure 3.

[0129] The computing device DV of figure 1 , configured to implement specific modules, is now described with reference to figure 3, in accordance with particular embodiments of the present invention. More specifically, the computing device DV may implements these modules by executing the computer program 12 stored in the memory 7.

[0130] The image retrieving module 110 is configured to retrieve an image capturing a handwriting input. This image, or image document, may be obtained by the image retrieving module 110 in any suitable manner. This image includes pixels of the captured handwriting input.

[0131] The image may represent handwritten content digitized as offline documents.

[0132] The handwriting input may be, for example, notes handwritten onto a notebook or a whiteboard and digitized by an imaging device such as a scanner or a camera.

[0133] The handwriting input may be a user input applied onto a touch-based user interface and captured as an image thus forming a static representation of the handwriting input.

[0134] The captured content may be saved as an image document on the capturing device and communicated with another device, for example a scanner or a smartphone communicating with the computing device DV.

[0135] The image document may be stored by the document management system for later use, e.g., by using the memory of the computing device DV.

[0136] The skeleton extracting module 120 is configured to extract the skeleton of pixels from the retrieved image. This skeleton may represent a trace of the handwriting input captured by the retrieved image.

[0137] In one embodiment, the skeleton extracting module 120 is (or comprises) a fully convolutional neural network (FCNN). The FCNN consists of a downsampling and an upsampling process. The downsampling process can be performed by an encoder, that encodes the input image in a small hidden feature map H. The upsampling process decodes the feature map H to the original resolution and outputs a target skeleton image.

[0138] In one embodiment, the skeleton extracting module 120 comprises a skeleton thinning module configured to thin down to a skeleton of unitary thickness by removing all the contour pixels of the image except those pixels that belong to the skeleton. The extracted skeleton may be further computed to only keep skeleton pixels with distinctive features of the handwriting trace. Skeleton thinning modules may be based on different criteria of the skeleton pixels as further explained in the publication of Stefanelli, R., & Rosenfeld, A. (1971 ), Some parallel thinning algorithms for digital pictures. Journal of the ACM (JACM), 18(2), 255-264.

[0139] In one example, the skeleton thinning module is configured to consider the binary digitized skeleton image as a matrix of pixels. Iterative transformations may be applied to the matrix of pixels according to the values of a set of neighboring pixels as further explained in the publication of Zhang, T. Y., & Suen, C. Y. (1984). A fast parallel algorithm for thinning digital patterns. Communications of the ACM, 27(3), 236-239.

[0140] The skeleton cutting module 132 is configured to cut (or divide) the skeleton into segments. The cutting is performed at junction pixels of the skeleton. This may be achieved by removing (therefore removing as foreground pixel and setting as background pixel) the different junction pixels JP and defining the resulting connected trace as segments SEG. In the binary digitized skeleton considered as a matrix of pixels, a junction pixel JP is defined as a skeleton pixel with 3 or more 8-connected skeleton pixels as shown on figure 4A. Each resulting segment will have two extremities. An extremity EX is the skeleton pixel with exactly one 8-connected skeleton neighbor (see fig. 4).

[0141] The sub-stroke generating module 134 is configured to generate a plurality of substrokes based on the segments determined by the skeleton cutting module 132. Each sub-stroke comprises a sequence of coordinates of a respective one of said segments, wherein each segment comprises two sub-strokes having opposite directions, the first sub-stroke starting at the first end and ending at the second end and the second substroke starting at the second end and ending at the first end.

[0142] The sub-stroke generating module 134 may compute a path or a sequence of ink points from one extremity of a segment to another to define a sub-stroke. The opposite traversal path may also be included as a distinct sub-stroke.

[0143] The embedding generating module 140 is configured to generate fixed-length embeddings representing a sub-stroke space such that it is informative both for reconstructing the original sub-strokes and for predicting the following sub-strokes. The embedding generating module 140 may be coupled with a sub-stroke vectorizer 142 configured to generate sub-stroke vectors and is coupled with an embeddingtransformer 144 configured to generate sub-stroke embeddings, as shown in figure 3.

[0144] In a particular example, the sub-stroke vectorizer 142 is configured to linearly project the sequence of ink points of a sub-stroke to a vector of a fixed first size, referred to as a sub-stroke vector SV, and to sum each ink point with a sinusoidal positional encoding. For example, the fixed first size of the sub-stroke vectors is 64.

[0145] Then, the sub-stroke vectors may be applied to an embedding-transformer 144. The embedding-transformer 144 generates sub-stroke second vectors of a fixed second size, referred to as sub-stroke embeddings SE. For example, the fixed second size of the sub-stroke embeddings is 8. The fixed second size defines a rather small dimensional embedding space which still allows to predict valid embeddings and improves the prediction performances.

[0146] The embedding-transformer 144 may be an encoder with a stack of six identical layers. Each layer may have (or comprise) two sub-layers. The first sub-layer may be a multihead self-attention mechanism and the multi-head attention may comprise four attention heads. The second sub-layer may be a fully connected feed-forward network.

[0147] In one embodiment, the embedding generating module 140 may be coupled with an embedding decoder 146. The embedding decoder 146 reconstructs a sub-stroke as a parametric approximation of an original sub-stroke. The decoder 146 may be a two- layer perceptron (MLP) comprising a hidden layer and an output layer producing the corresponding coordinates of a point of the reconstructed sub-stroke.

[0148] The embedding-transformer 144 coupled with the embedding decoder 146 may be an auto-encoder network. The auto-encoder network may be configured to reconstruct accurately the input sub-stroke. The auto-encoder may be trained independently from other modules described in the present invention. The auto-encoder may be trained according to any relevant loss function calculated between the reconstructed substroke and the original sub-stroke.

[0149] The sequence of sub-stroke reconstructing module 150 is configured to iteratively reconstruct the sequence of the sub-strokes by ordering the plurality of sub-strokes. In a particular example, the sequence of sub-stroke reconstructing module 150 is configured to:

[0150] - generate, based on the plurality of sub-stroke embeddings, a set of probabilities for each sub-stroke of a sub-stroke sequence, each probability for a given substroke defines the likelihood that another sub-stroke being the subsequent (or following) sub-stroke to the given sub-stroke in the sub-stroke sequence; and

[0151] - reconstruct the sequence of sub-strokes by selecting the sub-strokes with the highest probabilities;

[0152] In a particular example, the sequence of sub-stoke reconstructing module 150 is coupled with a subsequent sub-stroke probabilities calculating module as an ordering transformer 152, a subsequent sub-stroke determining module 154, and a sub-stroke concatenating module 156.

[0153] The subsequent sub-stroke probabilities calculating module as the ordering transformer 152, illustrated in figure 3, is configured to generate a probability of a pen state of a current sub-stroke, for example a pen-up state indicating if a current substroke is the end of the user original handwritten stroke, and extract a set of probabilities that each of the plurality of sub-strokes is the following sub-stroke of the current sub-stroke.

[0154] The ordering transformer may be an encoder-decoder transformer composed of a stack of multiple identical hidden layers 1520 and a last transformer decoder layer 1530.

[0155] Each of the multiple identical layers 1520 may comprise two sub-layers with a model size of dm and multiple attention heads.

[0156] The last transformer decoder layer 1530 may comprise three sub-layers including a masked multi-head attention 1532, a multi-head attention 1534 and a feed forward network 1536.

[0157] The multi-head attention 1534 computes cross-attention between current outputs of current query SEi processed by the preceding layers 1520 and a prior output of the embedding-transformer 144, wherein the output of the embedding-transformer 144 is considered as keys KYM and values VLM of a sub-stroke embeddings SEM . The multi-head attention 1534 computes attention scores over the plurality of substrokes which are extracted as a set of probabilities Pj+i (SSi ) ... Pi+i(SSn) that each of the plurality of sub-strokes is the subsequent (or following) sub-stroke of the current sub-stroke. The set of probabilities includes a probability Pi+i(eos) that the following sub-stroke is an theoretical end of the sub-stroke sequence.

[0158] The ordering transformer may be trained including a multi-layer perceptron (MLP) 153 to predict the pen-up state Pi(pen-up) of the current query SEi.

[0159] The ordering transformer may be trained with databases of handwritten input strokes stored as online data (pen trajectory) and offline data (digital images), wherein an online ink point can be mapped onto the corresponding scanned image, and conversely, each offline pixel can be temporally indexed.

[0160] In a particular embodiment, the multi-head attention sub-layer 1534 includes an attention refinement module (ARM) to refine the attention weight in the ordering transformer in order to alleviate the lack of coverage over the past predicted information.

[0161] The subsequent (or following) sub-stroke determining module 154 is configured to determine the subsequent (or following) sub-stroke of current sub-stroke query by selecting the sub-stroke with the highest probability of the set of probabilities. The subsequent sub-stroke determining module 154 is setting and feeding the determined subsequent sub-stroke as current query SEi to the ordering-transformer. When the determined following sub-stroke is set as the theoretical end of the sequence signal, the iterative process of the ordering transformer ends.

[0162] The sub-stroke concatenating module 156 is configured to reconstruct the sequence of sub-strokes by concatenating each following sub-stroke determined by the subsequent sub-stroke determining module 154 until the determination of the final state which ends the sequence.

[0163] The ink point sequence inferring module 160 is configured to infer the sequence of ink points of the captured handwriting input from the reconstructed sequence of substrokes. To this end, the ink point sequence inferring module 160 may use the predicted pen-up state information of each sub-strokes of the sequence. The sequence of ink points may be linearly interpolated to fill the void left between two consecutive 1 sub-stroke extremities when the pen state of the earlier sub-stroke is predicted as a pen down.

[0164] For the purpose of illustration, Figures 4A and 4B show an example of the reconstruction of a handwriting input according to a particular embodiment of the present system and method.

[0165] Figure 4A shows an image IM retrieved from an imaging device capturing a handwriting input as a pixel trace of one input stroke IN representing the French word “onze”.

[0166] For illustrative purposes on Figure 4A, the subsequent process is focusing on the sample image included in the window WD representing the letter “z” of the captured handwriting input, it is understood that the whole input stroke IN is processed according to the example of Figure 4A and 4B.

[0167] A skeleton SK of the handwriting input IN is shown as extracted from the image IN by a trained fully convolutional neural network (FCNN) and thinned to a 1 -pixel thick outline. Sampled skeleton pixels along the outline are represented as timestep points. The skeleton SK comprises fourteen junction pixels, numbered from 1 to 14 (figure 4A), wherein the junction pixels are connected to 3 or more other pixels in an image of a 8-connected pixels matrix representation. The fourteen junction pixels are removed (i.e excluded from the skeleton SK), resulting into splitting the skeleton SK into seven segments SG7 to SG13. The connected pixels of each segment may be labelled to obtain a sequence of points from one extremity to another and generate a sub-stroke SS101 defined by this point sequence. The opposite traversal path SS102 is also computed as a distinct sub-stroke, formed by the same ink points P1 -P27 but in the two opposite orders.

[0168] For illustrative purposes on Figure 4B, the subsequent process is only represented for the segment SG10, it is understood that all the segments SG1 to SG16 resulting from the split stroke SK may be processed according to the example figure 4B. The segment SG10 is shown as a sequences of points P1 to P27 resulting in two sub-strokes SS101 and SS102 of the two opposite directions of the sequence of points of the segment SG10. The sub-stroke SS101 is indexed from point P1 to point P27, i.e. according to a first order of ink points (from a first end P1 to a second end P27) within the sequence of the segment SG10, whereas the sub-stroke SS102 is indexed from point P27 to point P1 of the segment SG10, i.e. according to a second order (from the second end P27 to the first end P1 ), opposite to the first order, of ink points within the sequence of the segment SG10.

[0169] The timestep points (P1 to P27) of the sub-stroke SS101 may be linearly projected and summed, for instance with a sinusoidal positional encoding, for each timestep of the sub-stroke by the vectorizer 442 into a plurality of first vectors of a fixed first size.

[0170] The timestep points (P27 to P1 ) of the sub-stroke SS102 may be linearly projected and summed, for instance with a sinusoidal positional encoding, for each timestep of the sub-stroke by the vectorizer 442 into a plurality of first vectors of a fixed first size.

[0171] The plurality of first vectors may then be transformed through the embeddingtransformer 444 into a plurality of second vectors. The outputs SE101 and SE102 of respectively the last timestep points of each sub-stroke, for example P27 for SS101 , or P1 for SS102, are linearly projected to fixed size vectors resulting in the sub-stroke embeddings SE101 and SE102 respectively.

[0172] For illustrative purposes on Figure 4B, the subsequent process is only represented for the embedding SE102, it is understood that the embeddings resulting from the captured handwritten input stroke IN may be processed in a similar way. In the present example, the embedding SE102 is concatenated with the positional embedding of its starting point to add global information of the sub-stroke SS102 spatial arrangement in the image IM (not shown). The concatenated embedding SE102 is applied to the ordering-transformer 452 for outputting two decisions. The whole embeddings resulting from the processing of the input IN (SE11 to SE162) is transferred as output keys and values information (k11 / v11 to k162 / v162) to the ordering-transformer.

[0173] The first decision includes a set of probabilities PrS for the sub-stroke SS102 that each of the other sub-strokes (SS1 to SS162) is a following sub-stroke of SS102 in a sequence of sub-strokes SEQ. Additionally, the set of probabilities PrS includes a probability that the following sub-stroke is an end of the sequence (eos), therefore that the sub-stroke SS102 is the last sub-stroke of the sequence SEQ. The second decision is a probability that the sub-stroke SS102 is an end of the user original input stroke IN as represented on the figure 4A, referred to as a pen-up probability P(pen-up). The highest probability of the set of probabilities PrS predicts the following sub-stroke of the sequence Pi4(SS131 ), here shown as the fourteenth sub-stroke of the sequence of sub-strokes, wherein the thirteen first sub-strokes of the sequence have been predicted in earlier iteration of the process. The predicted following sub-stroke SS131 , resulting from the split of the segment SG13 shown in figure 4B is concatenated to the end of the reconstructed sequence of sub-strokes SEQ. Additionally, the predicted pen-state Pi3(pen-up) outcome of the ordering transformer 552 indicates a pen down (pen-up=0) that is the sub-stroke SS102 is connected to the predicted following substroke SS131 i4. in the reconstructed sequence SEQ.

[0174] The predicted following sub-stroke Pi4(SS131 ) is sent as indicator of the next substroke embedding to be fed to the ordering-transformer to compute the next iteration and predict the next following sub-stroke P15 (not shown).

[0175] The predicted sequence of sub-strokes is shown as starting from SS111 until SS13114 according to the predicted following sub-strokes of each given sub-stroke.

[0176] The predicted sequence of sub-strokes SEQ shown in figure 4B is resulting from the first fourteen iteration of the ordering transformer 452, predicting the probabilities P1 to P14 and concatenating the sub-strokes from the first indexed following sub-strokes SS111 (corresponding to the first direction of the sub-stroke SS11 generated from the first segment SG1 of the skeleton) to the latest (fourteenth) indexed following substroke SS13114 (corresponding to the first direction of the sub-stroke SS13 generated from the thirteenth segment SG13 of the skeleton).

[0177] For example in Figure 4B, the following sub-stroke of SS11 i is SS212. As another example on Figure 4B, the following sub-stroke of SS31s is SS324, wherein the reconstructed sequence predicts that the middle leg of the ‘n’ letter of the input stroke IN is traced descending according to SS31 and ascending according to SS32 over the same path of the same segment SG3.

[0178] The reconstructed sequence of sub-strokes SEQ advantageously allows to infer an ink point sequence, i.e. a temporal sequence of ink points, from the sequence of points of each sub-stroke of the sequence SEQ and the pen-state predicted at each iteration (1 to 16) wherein the prediction of a pen down indicates that the inferred sequence of ink points includes uninterrupted trace between the earlier sub-stroke of two consecutive sub-strokes as shown between the end of each sub-strokes of the sequence SEQ shown as little stars. The present invention thus advantageously allows recovering in an efficient and reliable manner temporal information of ink points comprised in handwriting, by processing a static representation (i.e. an image) of the handwriting. Based on a substroke level transformer approach, offline images of handwriting can be efficiently converted into an online representation of the handwriting. In particular, the invention may provide a sub-stroke transformer model that can efficiently reconstruct online signal (or online information) from offline images of handwriting. A pen trajectory can be reconstructed accurately to reflect the user’s offline handwriting.

[0179] The invention is based on a process moving from the image to a sequence framework to operate on the sub-stroke level and to perform a local and global analysis of the different junctions, thereby offering a performant online recovery system.

[0180] The temporal sequence recovered by the invention constitutes temporal information than can be advantageously used, for instance in an offline handwriting recognition system, to better analyze and process handwriting. A handwriting processing system can be made more accurate and responsive by using such temporal information, for instance to perform handwriting recognition on handwriting captured in an image. A user can thus leverage the best of the online and offline approaches, using for instance ergonomic note-taking with a pen and paper on the one hand, and powerful editing and processing of the digital ink on the other hand.

[0181] While not explicitly described, the present embodiments may be employed in any combination or sub-combination thereof. This disclosure having been described in particular embodiments, it is clear that it is susceptible to numerous modifications and embodiments are possible within the scope defined by the following claims.

Claims

Claims1 . Method for ink point sequence reconstruction, said method comprising:- retrieving (S110) an image (IM) depicting a captured handwriting input (IN), said image including pixels of the captured handwriting input;- extracting (S120) a skeleton (SK) of pixels from the retrieved image;- cutting (S132) the skeleton at junction pixels into a plurality of segments;- generating (S134) a plurality of sub-strokes based on said segments, each substroke comprising a sequence of coordinates of a respective one of said segments, wherein each segment comprises a first end and a second end and each segment comprises two sub-strokes having opposite directions, the first sub-stroke starting at the first end and ending at the second end and the second sub-stroke starting at the second end and ending at the first end;- generating (S140) a plurality of sub-stroke embeddings based respectively on the plurality of sub-strokes;- generating (S150), based on the plurality of sub-stroke embeddings, a set of probabilities for each sub-stroke of the plurality of sub-strokes, each probability for a given sub-stroke defines the likelihood that another sub-stroke being the subsequent sub-stroke to the given sub-stroke in a sequence of sub-strokes;- reconstructing (S160-S170) the sequence of sub-strokes by selecting the substrokes with the highest probabilities; and- inferring (S180) a temporal sequence of ink points of the captured handwritten input based on the reconstructed sequence of sub-strokes.

2. The method of claim 1 , wherein the generating (S150) of a set of probabilities for each sub-stroke of the sequence of sub-strokes comprises the following cycle:- setting a given sub-stroke embedding as an initial embedding;- generating (S150) a set of probabilities of sub-strokes being the subsequent sub-stroke of the given sub-stroke by applying the given sub-stroke embedding to an ordering-transformer;- determining (S160), as the subsequent sub-stroke of the given sub-stroke, the sub-stroke with the highest probability of the set of probabilities;- concatenating (S170) the determined subsequent sub-stroke to the end of the sequence of sub-strokes;- setting the given sub-stroke embedding as the embedding of the determined subsequent sub-stroke; wherein said cycle is repeated iteratively, for each given sub-stroke embedding, until the given sub-stroke embedding corresponds to the last sub-stroke of the sequence.

3. The method of claim 1 and 2, wherein one of the probabilities of the set of the probabilities corresponds to the end of the sequence, following the given substroke.

4. The method of any of claims 1 to 3, wherein the extracting of the skeleton is performed by applying the image to a fully convolutional neural network FCNN.

5. The method of any one of claims 1 to 4, wherein the cutting of the skeleton is performed by removing junction pixels wherein a junction pixel is a pixel from the skeleton with 3 or more 8-connected skeleton pixel neighbor.

6. The method of any one of claims 1 to 5, wherein the generating of a plurality of sub-stroke embeddings S140 from the plurality of respective sub-strokes comprises,- first, projecting the respective sequences of coordinates of the plurality of substrokes within a plurality of respective sub-stroke vectors;- then, transforming the plurality of sub-stoke vectors into the plurality of respective sub-stroke embeddings of by applying the plurality of sub-stroke vectors to an embedding-transformer.

7. The method of claim 6, wherein the plurality of sub-stroke vectors have a fixed first-size.

8. The method of claim 6 or 7, wherein the plurality of sub-strokes embeddings have a fixed second-size.

9. The method of any one of claims, 6 to 8 wherein the plurality of sub-stroke embeddings are reconstructed as a plurality of respective parametricapproximation of sub-stroke curves by an embedding-decoder, wherein the embedding-transformer is trained according to the reconstructed sub-strokes.

10. The method of any one of claims 1 to 8, wherein the generating a set of probabilities (S150) of a subsequent sub-stroke, comprises predicting a pen state for each given sub-stroke applied to the ordering-transformer.

11. The method of any one of claims 1 to 9, wherein the inferring of the ink point sequence based on the reconstructed sequence of sub-strokes comprises interpolating ink points between two consecutive sub-strokes when the pen state of the earlier sub-stroke is pen-down.

12. Computing device for ink point sequence reconstruction , said computing device comprising:- an image retrieving module (110) configured to retrieve an image depicting a captured handwriting input, said image including pixels of the captured handwriting input;- a skeleton extracting module (120) configured to extract a skeleton of pixels from the image;- a skeleton cutting module (132) configured to cut the skeleton at junction pixels into a plurality of segments;- a sub-stroke generating module (134) configured to generate a plurality of substrokes based on said segments, each sub-stroke comprising a sequences of coordinates of a respective one of said segments, wherein each segment comprises a first end and a second end and each segment comprises two substrokes having opposite directions, the first sub-stroke starting at the first end and ending at the second end and the second sub-stroke starting at the second end and ending at the first end;- an embedding generating module (140) configured to generate a plurality of sub-stroke embeddings based respectively on the plurality of sub-strokes;- a sub-stroke sequence reconstructing module (152-156) configured to: o generate, based on the plurality of sub-stroke embeddings, a set of probabilities for each sub-stroke of the plurality of sub-strokes, each probability for a given sub-stroke defines the likelihood that another sub-stroke being the subsequent sub-stroke to the given sub-stroke in a sequence of sub-strokes; and o reconstruct the sequence of sub-strokes by selecting the sub-strokes with the highest probabilities;- an ink point sequence inferring module (160) configured to infer a temporal sequence of ink points of the captured handwritten input based on the reconstructed sequence of sub-strokes.

13. The computing device of claim 12, wherein the sub-stroke sequence reconstructing module (150) comprises: o a sub-stroke vectorizer (142) configured to linearly project a sequence of ink points of a sub-stroke into a sub-stroke vector; and o an embedding-transformer (144), coupled to the sub-stroke vectorizer module (142), configured to generate a sub-stroke embedding from the respective sub-stroke vector.

14. The computing device of claim 12 or 13, wherein the sub-stroke sequence reconstructing module (150) comprises: o an ordering-transformer (152) configured to generate, for a sub-stroke, a probability of a pen state and extract a set of probabilities of a following sub-stroke; o a subsequent sub-stroke determining module (154), coupled to the ordering transformer (152), configured to determine the following substroke of a sub-stroke as the highest probability of the set of probabilities; and o a sub-stroke concatenating module (156), coupled to the subsequent sub-stroke determining module (154), configured to concatenate the determined following sub-stroke to the end of the sequence of substrokes.

15. The computing device of any one of claims 12 to 14 further comprising: a transformer decoder configured to reconstruct the plurality of sub-stroke as a plurality of respective parametric approximation of the sub-stroke.

Images