Systems and methods for real-time end-to-end capturing of ink strokes from video

Abstract

A real-time end-to-end system for capturing ink strokes written with ordinary pen and paper using a commodity video camera is described. Compare to traditional camera-based approaches, which typically separate out the pen tip localization and pen up / down motion detection, described is a unified approach that integrates these two steps using a deep neural network. Furthermore, the described system does not require manual initialization to locate the pen tip. A preliminary evaluation demonstrates the effectiveness of the described system on handwriting recognition for English and Japanese phrases.

Description

BACKGROUND OF THE INVENTIONTechnical Field[0001]The disclosed embodiments relate in general to systems and methods for data entry and, more specifically, to systems and methods for real-time end-to-end capturing of ink strokes from video.Description of the Related Art[0002]Data entry with a pen and paper can form a natural and convenient interface between a computer and a user. While the keyboard is the main input device to a computer, people still tend to prefer handwritten input over the keyboard in many tasks, such as note taking, brainstorming in a meeting, and hand sketch-based modeling. While there exist commercial products that can record ink strokes, they require special pen and / or paper with printed patterns, as described, for example in U.S. Pat. No. 7,570,813. Examples of such commercial products include: FX Denshi Pen Fuji Xerox Denshi-Pen, Anoto Livescribe Pen, Wacom Bamboo Spark, and the recently introduced Moleskin Smart Writing Set. These can be useful for vertical a...

AI Technical Summary

Benefits of technology

The patent describes a system and method for recognizing handwritten ink strokes using a neural network. The system includes a camera for capturing video of the handwriting and a processing unit for processing the captured video. The neural network detects the ink strokes and recognizes the handwriting using feature representations learned from convolutional network blocks. The system can be used in a variety of settings, such as at a desk or on a computer. The technical effects of the invention include improved accuracy and efficiency in recognizing handwritten ink strokes and text.

Problems solved by technology

Hence, most of the existing camera-based approaches still fall short of real-time performance and not applicable to applications where recognition needs to be performed at the time of writing.

Images