System for recognizing online handwriting

Abstract

The invention relates to a system (1) for recognizing online handwriting, comprising: - a handwriting instrument (2) comprising a main body (3) extending longitudinally between a first end (4) and a second end (5), the first end (4) having a writing tip (6) capable of writing on a support, the handwriting instrument (2) further comprising a module (17) comprising at least one movement sensor (7) configured to acquire movement data relating to the user's handwriting as the user is writing a sequence of characters using the handwriting instrument (2), - a computing unit (8) in communication with the at least one movement sensor (7) and configured to analyze the movement data by means of a machine learning model trained in a multitask manner so that it is capable of simultaneously performing at least two tasks, the machine learning model being configured to deliver as output the sequence of characters written by the user using the handwriting instrument.

Description

[0001] This application claims priority to European patent application EP20306281.5, filed on 26 October 2020, with respect to claims 1-8, the contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of systems for recognizing online handwriting. More specifically, this disclosure relates to the field of systems for online and continuous handwriting recognition. Background Technology

[0003] Currently, several systems or methods exist for recognizing users' handwriting. One type of handwriting recognition, known as online handwriting recognition, consists of recognizing characters or character sequences while the user is writing them. Another type of handwriting recognition, known as offline handwriting recognition, is based on the analysis of images displaying text.

[0004] For example, patent document US10126825 discloses a method for performing online handwriting recognition using a device such as a mobile terminal and three sensors: an accelerometer, a gyroscope, and a magnetometer. The method described in this document uses a bidirectional long short-term memory (BLSTM) type neural network. Data acquired by the three sensors is submitted to several preprocessing steps for analysis by the neural network. Furthermore, the neural network is trained using a dictionary containing predetermined words. The neural network is trained to recognize the beginning and end of the words being written, and based on this, it can determine which word in the dictionary has been written. One drawback of this method is the use of three sensors, which results in high device construction costs. Another drawback of this document is that the neural network is trained to recognize predetermined words; the recognition is not continuous.

[0005] Also from the scientific publications by Xie et al. published in the August 2018 issue of IEEE Transactions on Pattern Analysis and Machine Intelligence. Spatial semantic context is learned through a fully convolutional recurrent network for online handwritten Chinese text recognition. ( Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text RecognitionA method for detecting online handwriting is known. This method uses four neural networks: Multi-Spatial Context Fully Convolutional Recurrent Network (MC-FCRN), Fully Convolutional Network (FCN), Long Short-Term Memory (LSTM), and Connectionist Temporal Classification (CTC). The neural networks are jointly trained. The method uses a path signature method applied to a reduced data window. This publication does not describe how to efficiently acquire the data, but using a path signature method implies the use of a position sensor, such as a camera or touchscreen. Another drawback is the need to use four neural networks, each trained in a chain with entries from the previous neural network.

[0006] Also from the publication by Liu et al. in ArXiv in 2020. Offline hand using convolutional neural networks Chinese text recognition (Offline Handwritten Chinese Text Recognition with Convolutional Neural Networks) A method for offline handwriting recognition is known, which segments a text line image into small time steps and then feeds these small time steps into a feature extractor neural network. However, this publication performs offline handwriting recognition, which is not suitable for online handwriting recognition. Summary of the Invention

[0007] One objective of this disclosure is to improve this situation.

[0008] A system for recognizing online handwriting is proposed, comprising:

[0009] - A handwriting device comprising a body extending longitudinally between a first end and a second end, the first end having a writing tip capable of writing on a support; the handwriting device further comprising a module including at least one motion sensor configured to acquire motion data about the user's handwriting while the user is writing a sequence of characters using the handwriting device.

[0010] - A computing unit that communicates with at least one motion sensor and is configured to analyze the acquired motion data through a machine learning model trained in a multi-task manner, the machine learning model being configured to deliver as output a sequence of characters written by the user using a handwriting device.

[0011] In one embodiment, the module is embedded in the second end of the handwriting device.

[0012] In this embodiment, the module differs from the handwriting device; that is, the module is a separate component. The module can be placed on the outer surface of the second end of the handwriting device.

[0013] In one embodiment, the module further includes a computing unit.

[0014] In one embodiment, the module further includes a short-range radio communication interface configured to transmit raw motion data acquired by the motion sensor to the mobile device, which includes a computing unit, via the communication interface of the mobile device.

[0015] In this embodiment, the motion sensor is a triaxial accelerometer.

[0016] In one embodiment, the module further includes a second motion sensor, which is a three-axis gyroscope.

[0017] In this embodiment, the characters in the character sequence include numbers and / or letters.

[0018] The system disclosed herein can be used in a method for recognizing online handwriting, the method comprising:

[0019] - Motion data about the user's handwriting is acquired via a handwriting device while the user is writing a sequence of characters. The handwriting device includes a module comprising at least one motion sensor, and further includes a body extending longitudinally between a first end and a second end, the first end having a writing tip capable of writing on a support.

[0020] - The acquired motion data is analyzed using a machine learning model trained in a multi-task manner. The machine learning model is configured to deliver a sequence of characters written by the user using a handwriting device as output.

[0021] In an embodiment, the method further includes a prior step of multi-task training of a machine learning model, wherein the machine learning model is trained to perform:

[0022] - Stroke segmentation task.

[0023] -Character classification task.

[0024] In an embodiment, the stroke segmentation task includes labeling samples of the acquired motion data in at least one of the following categories:

[0025] - Drawings on paper

[0026] - Moving in the air.

[0027] In this embodiment, the acquired motion data is preprocessed by the following steps before being used in the stroke segmentation task:

[0028] - Window the raw motion data in N time frames, each sample in which is labeled in one of a pen drawing on paper and / or a movement in the air.

[0029] In this embodiment, the airborne movement marker includes at least two sub-markers:

[0030] -Move forward into the air,

[0031] - Move backward into the air.

[0032] In one embodiment, once trained, the machine learning model is stored, and the method further includes continuously and in real-time analyzing the acquired motion data using the trained machine learning model. Attached Figure Description

[0033] Other features, details, and advantages will be shown in the following detailed description and accompanying drawings, in which:

[0034] Figure 1

[0035] [ Figure 1 The illustration shows a system for detecting handwriting problems according to a first embodiment.

[0036] Figure 2

[0037] [ Figure 2 ]exhibit Figure 1 The system block diagram shown is shown.

[0038] Figure 3

[0039] [ Figure 3 The illustration shows a system for detecting handwriting problems according to a second embodiment.

[0040] Figure 4

[0041] [ Figure 4 ]exhibit Figure 3 The system block diagram shown is shown.

[0042] Figure 5

[0043] [ Figure 5 The illustration shows a system for detecting handwriting problems according to a third embodiment.

[0044] Figure 6

[0045] [ Figure 6 ]exhibit Figure 5 The system block diagram shown is shown.

[0046] Figure 7

[0047] [ Figure 7 This diagram illustrates the multi-task training phase of a machine learning model according to an embodiment.

[0048] Figure 8

[0049] [ Figure 8 This diagram illustrates the inference phase of a machine learning model following its multi-task training, according to an embodiment.

[0050] Figure 9A and 9B

[0051] [ Figure 9A and 9B It displays a sequence of characters written by the user using a handwriting device, along with associated motion data. Detailed Implementation

[0052] Now for reference Figures 1 to 6 An embodiment of system 1 for recognizing online handwriting is shown. The same reference numerals are used to describe the same elements of system 1.

[0053] Figure 1 and 2 System 1 according to a first embodiment is generally shown. System 1 includes a handwriting device 2. The handwriting device 2 may be a pen, pencil, paintbrush, or any element that allows the user to write on a support. The support may be paper, canvas, or any surface on which the user can write or draw. In embodiments, the support may be any non-electronic surface.

[0054] The handwriting instrument 2 includes a body 3 extending longitudinally between a first end 4 and a second end 5. The first end 4 includes a writing tip 6 capable of writing on a support. The tip 6 can deliver ink or color.

[0055] System 1 may include module 17. In an embodiment, module 17 includes at least one motion sensor 7. In an embodiment, motion sensor 7 may be a triaxial accelerometer.

[0056] In an embodiment, module 17 may further include a second motion sensor 7. For example, the second motion sensor 7 is a three-axis gyroscope.

[0057] In this embodiment, module 17 is embedded in the handwriting device 2. For example, module 17 is embedded near or at the second end 5 of the handwriting device.

[0058] In this embodiment, module 17 differs from handwriting device 2. Figure 3 and 4 In the illustrated embodiment, module 17 can be placed on the outer surface of the second end 5 of the handwriting device.

[0059] Therefore, module 17 may include a hole with a diameter D to allow the second end 5 of the handwriting instrument to be inserted therein.

[0060] In one embodiment, at least one motion sensor 7 is then placed or located at the second end 5 of the handwriting device 2. In this way, the user of the handwriting device will not be disturbed when using the handwriting device 2, because the module 17 will not obstruct their view.

[0061] For all embodiments described below, module 17 may be embedded in the handwriting device 2, or may be different from / separate from the handwriting device 2.

[0062] In one embodiment, at least one motion sensor 7 is capable of acquiring data about the user's handwriting while the user is using the handwriting device 2. This data is transmitted to a computing unit 8, which is configured to analyze the data and perform handwriting recognition.

[0063] The computing unit 8 may include a volatile memory for storing data acquired by the motion sensor 7 and a non-volatile memory for storing a machine learning model for handwriting recognition.

[0064] exist Figures 1 to 4 In the embodiment shown, module 17 further includes a computing unit 8.

[0065] exist Figure 5 and 6 In the embodiments shown, module 17 does not include computing unit 8.

[0066] In this embodiment, the computing unit 8 may be included in a mobile device. The mobile device 11 may typically be a tablet computer, a mobile phone, or a computer.

[0067] Module 17 may further include a short-range radio communication interface 9 that allows data communication between at least one motion sensor 7 and the computing unit 8. In embodiments, the short-range radio communication interface uses Wi-Fi, Bluetooth®, LoRa®, SigFox®, or NBIoT networks. In embodiments, the short-range radio communication interface may also use 2G, 3G, 4G, or 5G networks for communication.

[0068] The mobile device 11 further includes a short-range radio communication interface 12 that enables communication between the computing unit 8 and the module 17 via a short-range radio communication interface 9.

[0069] In this embodiment, the computing unit 8 may be included in the cloud. The computing unit 8 may be accessed via a mobile device 11.

[0070] When computing unit 8 is included in module 17, such as Figures 1 to 4 As shown, module 17 may still include a short-range radio communication interface 9 to transmit results to the expelled server (not shown), as a non-limiting example.

[0071] Module 17 may further include a battery 10 that supplies power to at least the motion sensor 7 when the user is using the handwriting device. The battery 10 may also supply power to all other components that may be included in module 17, such as the computing unit 8 and the short-range radio communication interface 9.

[0072] In all embodiments, the computing unit 8 receives motion data acquired from at least one motion sensor 7 to analyze the motion data and perform handwriting recognition.

[0073] In this embodiment, the motion data received by the computing unit 8 is the raw motion data.

[0074] More specifically, the computing unit 8 may store a machine learning model capable of analyzing motion data acquired by the motion sensor 7. The machine learning model may include at least one neural network. The machine learning model is trained to perform the recognition of character sequences written by the user while using the handwriting device 2.

[0075] Train machine learning models using a multi-task approach.

[0076] More specifically, the machine learning model can be trained to perform two tasks: the recognition of characters written by the user while using the handwriting device 2, which is denoted as the first task, and the segmentation of strokes, which is denoted as the second task.

[0077] The first and second tasks share the same backbone of the neural network, which extracts hidden features from motion data. In a non-limiting embodiment, this neural network is a convolutional neural network, such as an encoder. This encoding neural network is hereinafter referred to as sharing a backbone.

[0078] In one embodiment, the first task can be performed by a neural network serving as a shared backbone, together with a recurrent neural network trained using the CTC loss function, such as a BLSTM or a converter.

[0079] In one embodiment, the second task can be performed by an autoencoder neural network, wherein the encoder of the autoencoder neural network is a shared backbone and the decoder is an additional convolutional neural network with upsampling.

[0080] Train the machine learning model in a multi-task manner so that it can perform at least two tasks simultaneously.

[0081] refer to Figure 7 This illustrates the steps in the training phase of a multi-task machine learning model.

[0082] In step S1, the computing unit 8 receives motion data acquired by at least one motion sensor 7 while the user is writing characters or character sequences using the handwriting instrument 2.

[0083] Perform preprocessing step S2. The preprocessing step includes temporal windowing of the motion data, which divides the motion data into time steps.

[0084] In this example, the motion data is windowed into time steps ranging from 1 second to 5 seconds.

[0085] Next, windowed motion data, i.e., time steps, are fed into the shared backbone. In step S3, the shared backbone further receives tags.

[0086] In this embodiment, the markings correspond to both on-paper and airborne time stamps. In other words, the markings correspond to whether the writing tip 6 of the handwriting device 2 touches the paper (on-paper marking) or not (airborne marking). The markings are time-stamped and relative to each data sample.

[0087] Typically, when using a motion sensor 7, such as a three-axis accelerometer (or a three-axis accelerometer and a three-axis gyroscope), the motion data acquired by the motion sensor 7 is continuous and lacks positional information. More specifically, the motion data includes both motion data corresponding to when the user is writing on the support using the writing instrument and motion data corresponding to when the user is merely moving the writing instrument 2 in the air without writing.

[0088] At step S5, the shared backbone performs hidden feature extraction to obtain intermediate features at step S6.

[0089] Based on these intermediate features, at step S7, a segmentation task or a second task is performed. This step allows for the discovery of the start and end times of each stroke. During this step, the segmentation task includes segmentation of strokes on paper and strokes moving in the air.

[0090] In step S8, paper strokes and air movement categories are obtained for each time sample in the signal.

[0091] The intermediate features from step S6 are also used in step S9. In step S9, the machine learning model performs concatenation of the time series of pen drawings on paper and aerial movements.

[0092] In step S10, other tags are sent to the machine learning model for the first task of character sequence recognition.

[0093] More specifically, these labels can be fed into the recurrent neural network involved in the first task.

[0094] These tags may correspond to the character tags or character sequence tags used in the character sequence classification step S11.

[0095] Tags can be individual characters, words, or even sentences.

[0096] The classification step S11, as described above, can be performed at least in part by using a recurrent neural network trained with a loss function such as the CTC loss function, such as a BLSTM or a converter.

[0097] Finally, a character sequence is obtained at step S12, which corresponds to the character sequence written by the user when using the handwriting device 2, and the motion data at step S1 is associated with the character sequence.

[0098] During the training of the machine learning model, in addition to continuously recognizing character sequences, the machine learning model also learns to recognize both pen strokes on paper and movement in the air from motion data.

[0099] Multi-task training of machine learning models makes the system more efficient. In fact, learning the second task and the first task simultaneously makes learning the first task significantly more efficient.

[0100] Furthermore, because machine learning models acquire knowledge from motion data to identify individual strokes on paper and movements in the air, they are able to learn any character in any language.

[0101] Steps S1 to S6 and S9 to S12 can be assimilated into the first task of character sequence recognition.

[0102] Steps S1 to S8 can be assimilated into the second task of stroke segmentation.

[0103] Once multi-task training is performed, the machine learning model can be stored in computing unit 8.

[0104] In one embodiment, the weights of the first task are used only during the inference phase, such that the machine learning model stored in the computing unit includes only the weights of the first task.

[0105] In this embodiment, the machine learning model is further trained to distinguish between forward and backward air movements. In reality, pen and paper strokes are not always ordered, making it impossible to assume that the most recent stroke belongs to the most recent character, as one can go back to complete the previous character.

[0106] This is shown in Figure 9A and 9B Above, the sequence of “written down” characters written on paper and the associated motion data are displayed.

[0107] In this example, the motion data corresponds to the acceleration signal acquired by the triaxial accelerometer. Figure 9B The graphic display shows the amplitude of the acceleration signals of the tree axes x, y, and z as they change over time.

[0108] Figure 9AThe character sequence consists of ten strokes, six forward jumps between letters, one forward jump between words, and two backward jumps (before the dot "i" and before the horizontal line "t".

[0109] The circled numbers are associated with each of the pen and paper drawings and show the order in which the pen and paper drawings were written.

[0110] Therefore, it is also important to consider the direction of movement in the air.

[0111] Specifically, forward air movement detection helps machine learning models identify the space between characters or words. Backward air movement detection helps machine learning models identify when a character is being completed (e.g., a dot on an 'i').

[0112] Next, in Figure 7 In step S3, the marker may include a paper time marker, forward air movement, and backward air movement, so that in step S7, a segmentation task is performed to obtain the paper strokes, forward air movement, and backward air movement.

[0113] In one embodiment, another marker can be used to train the machine learning model. This marker can be other aerial movements.

[0114] This marker helps the machine learning model determine when the user is no longer writing. For example, the machine learning model detects that the user is thinking or simply shaking the handwriting device 2 in the air.

[0115] The trained machine learning model can then be stored in computing unit 8.

[0116] Figure 8 This illustrates the inference phase of a machine learning model.

[0117] In step S20, motion data is obtained from at least one motion sensor 7.

[0118] In this example, the motion data corresponds to the raw signal acquired by at least one motion sensor 7 when the user writes a sequence of characters using the handwriting device 2.

[0119] Next, in step S21, the motion data is preprocessed by windowing these motion data into time steps. This step corresponds to the reference... Figure 7 Step S2 is described.

[0120] Next, in step S22, windowed motion data is fed into the shared backbone, which performs, for example, reference... Figure 7 The feature extraction described in step S5.

[0121] Based on these feature extractions, the machine learning model performs concatenation according to the time series, so that the character sequence classification performed in step S24 is performed on the paper strokes and air movements in the correct time order.

[0122] As described, air movement can include forward air movement and backward air movement.

[0123] In step S25, a character sequence is obtained as the output of a machine learning model, the output corresponding to a character sequence written by the user using a handwriting device.

[0124] In this example, the machine learning model no longer performs the segmentation task during the inference phase because, due to multi-task training, the shared backbone has been trained with weights corresponding to both the segmentation and character classification tasks.

[0125] This disclosure is not limited to the embodiments described herein. Other variations may be implemented.

[0126] In the embodiments, the machine learning model is used in the segmentation task ( Figure 7 Following step S7), stroke acceleration regression and / or air movement acceleration regression tasks are performed. This allows the machine learning model to derive the coordinates of the writing tip based on motion data. In this way, the machine learning model can deliver a copy of the character sequence written by the user (containing the shape of the characters) as output.

Claims

1. A system (1) for recognizing online handwriting, comprising: - A handwriting instrument (2) comprising a body (3) extending longitudinally between a first end (4) and a second end (5), the first end (4) having a writing tip (6) capable of writing on a support, and a module (17) including at least one motion sensor (7), the motion sensor being a three-axis accelerometer and / or a three-axis gyroscope, the motion sensor being configured to acquire motion data about the user's handwriting while the user is using the handwriting instrument (2) to write a sequence of characters. - A computing unit (8), which communicates with the motion sensor (7) and is configured to analyze the motion data through a machine learning model trained in a multi-task manner, enabling it to simultaneously perform stroke segmentation and character classification tasks. The machine learning model is configured to deliver the character sequence written by the user using the handwriting device as output, wherein: - The machine learning model includes a convolutional neural network that performs hidden feature extraction to obtain intermediate features, wherein the convolutional neural network is a multi-space context fully convolutional recurrent network MC-FCRN or a fully convolutional network FCN; - The stroke segmentation task is performed using an additional convolutional neural network with upsampling based on intermediate features to segment strokes on paper and movement in the air. The additional convolutional neural network is a multi-spatial context fully convolutional recurrent network MC-FCRN or a fully convolutional network FCN with upsampling. - The character classification task is performed using intermediate features with a recurrent neural network, which is a BLSTM or converter trained using the connectionist time classification CTC loss function.

2. The system (1) according to claim 1, characterized in that, The module (17) is embedded in the second end (5) of the handwriting instrument.

3. The system according to claim 1, characterized in that, The module (17) is placed on the outer surface of the second end (5) of the handwriting instrument (2).

4. The system according to claim 1, characterized in that, The module (17) further includes the computing unit (8).

5. The system (1) according to claim 1, characterized in that, The module (17) further includes a short-range radio communication interface (9), which is configured to transmit raw motion data acquired by the motion sensor (7) to the mobile device (11) including the computing unit (8) via the communication interface (12) of the mobile device (11).

6. The system (1) according to claim 1, characterized in that, The characters in the character sequence include numbers and / or letters.

7. The system (1) according to claim 1, characterized in that, The computing unit (8) includes a volatile memory for storing data acquired by the motion sensor.

8. The system (1) according to claim 1, characterized in that, The computing unit (8) includes non-volatile memory to store the machine learning model that implements handwriting recognition.

9. The system (1) according to claim 1, wherein the trained machine learning model is stored in the computing unit (8).

10. The system (1) according to claim 9, characterized in that, The stroke segmentation task includes labeling the acquired motion data in at least one of the following categories: strokes on paper or movement in the air.

11. The system (1) according to claim 9, characterized in that, The trained machine learning model is further configured such that the acquired motion data is preprocessed before being used in the stroke segmentation task, wherein the preprocessing includes windowing the raw motion data in time frames of N samples, wherein the samples are labeled in one of the strokes on paper or the movement in the air.

12. The system (1) according to claim 10, characterized in that, An air movement marker includes at least two sub-markers: forward air movement and backward air movement.

13. The system (1) according to claim 9, characterized in that, The trained machine learning model is further trained to enable it to perform the two tasks simultaneously.

14. The system (1) according to claim 1, characterized in that, The module (17) further includes a battery (10) configured to provide power to at least the motion sensor (7) when the user is using the handwriting device.

15. The system (1) according to claim 14, characterized in that, The battery (10) is further configured to provide power to other components included in the module (17).

16. The system (1) according to claim 15, characterized in that, The battery (10) is further configured to provide power to the computing unit (8) or the short-range radio communication interface (9).

Images