Electronic handwriting pen device based on inertial sensor and handwriting recovery method and system

Abstract

The invention belongs to the field of information technology and man-machine interaction, and relates to an electronic handwriting pen device based on an inertial sensor and a handwriting recovery method and system. The electronic handwriting pen device comprises a handwriting pen and an IMU (Inertial Measurement Unit) sensor, wherein the IMU sensor is fixed on a pen body of the handwriting pen. The handwriting recovery method comprises the steps that an electronic handwriting pen device based on an inertial sensor is adopted to collect data frames of an IMU sensor in real time, and a data sequence is generated; extracting time domain features and frequency domain features according to the generated data sequence, detecting a pen event according to the extracted time domain features and frequency domain features, identifying a writing state and dividing data intervals; calculating the stroke track of the pen falling interval and the stroke displacement of the pen lifting interval; and performing handwriting recovery according to the stroke track of the pen falling interval and the stroke displacement of the pen lifting interval. Original writing experience is reserved to a large extent, the recovery process is automatically carried out, a user does not need to intervene in operation, the smooth recovery effect can be achieved, and the method has good actual popularization value.

Description

Technical Field

[0001] The present invention belongs to the fields of information technology and human-computer interaction, and particularly relates to an electronic stylus device based on an inertial sensor, as well as a handwriting restoration method and system. Background Art

[0002] Pen input is an important way for people to record information and generate text content. At present, pen-and-paper writing is the most natural and convenient writing method. Users write with a stylus on a paper surface, and the handwriting is recorded on the corresponding paper. With the development of intelligent touch devices and human-computer interaction technologies, a batch of input devices that can collect electronic handwriting information have emerged, such as tablet computers, digitizing tablets, etc. Users write with a specific electronic pen on a matching planar device, and the device collects information such as the position, pressure, and tilt of the pen tip on the plane in real time and converts it into electronic handwriting. This electronic handwriting is different from the handwriting produced by traditional pen-and-paper writing. It is stored in an electronic device in electronic form. Therefore, with the assistance of interactive software, users can drag, scale, erase, rotate it, and modify colors, thicknesses, etc., greatly expanding the interaction space between users and handwriting, improving the efficiency of users in editing and creating handwriting, and promoting the realization of tasks such as intelligent typesetting, handwriting beautification, automatic handwriting generation, and style transfer. It has a wide range of applications in industries such as office, education, and medical care.

[0003] Currently, products on the market for collecting electronic handwriting, such as the Surface computer based on a capacitive screen and an electronic pen, and the Anoto Pen based on dot matrix paper and a micro camera. However, these products limit users to writing only on specific surfaces, not only restricting the writing range but also making the device less convenient to carry. The writing experience on different electronic devices is also reduced to varying degrees compared to pen-and-paper writing. In existing research work, researchers have tried to use various sensors to achieve writing on free surfaces, such as using an optical flow sensor to transform the pen tip into a form similar to a mouse; using an infrared detection device to place a base under the writing hand to detect the position and movement of the pen tip within a certain area on the writing plane; using a magnetic sensor to install a magnetic sensing array under the writing plane to sense the position and attitude of a pen with a magnet above the desktop; using a sonar device to place sonar transceiver equipment on the desktop to detect the movement of a pen with a speaker within a spatial range. These methods sacrifice the comfort and convenience of writing to varying degrees. Therefore, an electronic handwriting acquisition and restoration method close to the daily writing experience of users is still a research topic. Different from the existing work, the present invention uses an inertial sensor to achieve handwriting restoration. By fixedly installing an inertial sensor on an ordinary stylus, sensor data is collected and processed during user writing to achieve real-time restoration of the handwriting written by the user on a plane. Not only is the device simple, only requiring one sensor, which can avoid a complex device installation process, but also it can retain the natural experience of ordinary pen-and-paper writing. Summary of the Invention

[0004] The object of the present invention is to provide an electronic stylus device based on an inertial sensor, as well as a handwriting restoration method and system.

[0005] The inertial sensor in the present invention is an IMU (Inertial Measurement Unit), which is a sensor that can collect motion information in real time. The information collected includes but is not limited to three-axis acceleration and three-axis angular velocity. By fixing the IMU sensor on an ordinary stylus, the present invention collects the motion data generated by the sensor during the writing process in real time, identifies the pen-down and pen-up events through machine learning methods, and restores the pen-down strokes and pen-up displacements through deep learning methods, aiming to solve the problems that existing methods require the installation of external devices or the use of special pens, and damage the natural writing experience, and is not limited to specific IMU or stylus brands or models.

[0006] The technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, the present invention provides an electronic stylus device based on an inertial sensor. The device includes a stylus and an IMU sensor, and the IMU sensor is fixed on the body of the stylus.

[0008] Further, the outer shell of the stylus is made of a hard material, and the body of the stylus is designed as a non-deformable structure; the stylus and the IMU are rigidly connected, and there will be no relative movement between the two during use.

[0009] In some embodiments, the installation direction of the IMU sensor is set as follows: the x-axis of the IMU sensor is parallel to the body of the stylus and points to the tip direction, and the y-axis and z-axis of the IMU sensor are perpendicular to the body of the stylus.

[0010] In some embodiments, the IMU sensor is fixed on the side of the body of the stylus, which can avoid the inconvenience of use or carrying caused by the overlong body of the stylus. The IMU can be fixed at the tail end, the front end or other positions of the body of the stylus, and can be determined according to the actual situation such as the shape of the body of the stylus and the hand-holding posture, without interfering with the normal use of the electronic pen.

[0011] In some embodiments, the IMU sensor sends the acquired data to the host computer for processing, which can utilize the high computing power and storage advantages of an external computer, avoid the constraints of the volume and power consumption of the computing module on the electronic pen device, and improve the effect of handwriting restoration. The IMU can communicate with the host computer in a wired or wireless manner, which can be determined according to the design method of the IMU and the user's needs.

[0012] In a second aspect, the present invention provides a handwriting restoration method based on an inertial sensor, and its steps include:

[0013] Using the above-mentioned electronic stylus device based on inertial sensors of the present invention, the data frames of the IMU sensors are collected in real time and data sequences are generated;

[0014] Time-domain features and frequency-domain features are extracted according to the generated data sequences, pen events are detected according to the extracted time-domain features and frequency-domain features, and then the writing state is identified and data intervals are divided. The data intervals include a pen-lifting interval and a pen-down interval;

[0015] The stroke trajectory of the pen-down interval and the stroke displacement of the pen-lifting interval are calculated;

[0016] Based on the stroke trajectory of the pen-down interval and the stroke displacement of the pen-lifting interval, handwriting restoration is performed.

[0017] Further, the extracting time-domain features and frequency-domain features according to the generated data sequences, detecting pen events according to the extracted time-domain features and frequency-domain features, and then identifying the writing state and dividing data intervals includes:

[0018] The data of each channel of the IMU sensor are filtered, and fusion calculations are performed on the acceleration, angular velocity, and other available data to obtain the real-time attitude information of the IMU sensor;

[0019] Based on the obtained attitude information, the data of each channel are rotated from the sensor body coordinate system to the ground coordinate system, and the influence of gravity is eliminated;

[0020] Based on the above data sequences, a sliding window with a fixed length is set to record the latest data generated through the window;

[0021] The time-domain features and frequency-domain features of the data in the window are extracted for pen event recognition;

[0022] A pen event classifier based on a machine learning model is used to classify the time-domain features and frequency-domain features of the data in the current window to detect pen events, and the current writing state is updated according to the classification results, and data intervals are divided.

[0023] Further, the time-domain features include but are not limited to the sum, average, maximum, minimum, variance, median, peak, and skewness of the data values of each channel.

[0024] Further, the frequency-domain features include but are not limited to the peak, skewness, difference between the maximum and minimum values of the energy of the data values of each channel, the peak, skewness, and difference between the maximum and minimum values of the short-time Fourier transform values.

[0025] Further, the pen events include a pen-lifting event and a pen-down event, the writing states are divided into a pen-lifting state and a pen-down state, and the data intervals are divided into a pen-lifting interval and a pen-down interval.

[0026] Further, the pen event classifier employs, but is not limited to, a support vector machine.

[0027] Further, when a pen-down event or a pen-up event occurs, data intervals are partitioned. Energy features of data in a window are employed, but are not limited to, and the data frame with the maximum energy peak is selected as the boundary of the data interval. Let the data before this data frame be the data in the original writing state, and the data from this data frame onwards be the data in the updated writing state.

[0028] Further, calculating the stroke trajectory in the pen-down interval and the stroke displacement in the pen-up interval includes:

[0029] Rotating the data of each channel from the ground coordinate system to the writing surface coordinate system according to a predefined writing direction;

[0030] Setting three data buffers to record the data in the latest three generated data intervals;

[0031] When a pen-up event is recognized, normalizing and resampling the data in the three buffers to prepare input data for a further handwriting restoration model;

[0032] Inputting the normalized and resampled data sequence into an encoder, extracting temporal features in the data through spatio-temporal modeling, and generating a hidden layer feature vector;

[0033] Inputting the hidden layer feature vector into a decoder to calculate the stroke trajectory corresponding to the pen-down interval;

[0034] Inputting the stroke trajectory in the pen-down interval into a stroke encoder to extract stroke features in the data through spatio-temporal modeling;

[0035] Inputting the stroke features in the pen-down interval and the stroke trajectories in the three intervals into a displacement decoder to calculate the stroke displacement in the pen-up interval.

[0036] Further, the stroke represents the trajectory generated by a continuous movement of the pen tip on the writing plane, and the stroke displacement in the pen-up interval represents the displacement from the end of the previous stroke to the beginning of the next stroke.

[0037] Further, the encoder employs, but is not limited to, a recurrent neural network.

[0038] Further, the decoder employs, but is not limited to, a fully connected network.

[0039] In a third aspect, the present invention provides a handwriting restoration system based on an inertial sensor, including the following modules:

[0040] An IMU sensor module is used to collect the data generated by the IMU during the writing process in real time and generate a data sequence. Based on the data of each channel, attitude filtering and coordinate system rotation are performed to obtain the original data sequence.

[0041] A writing state classification module is used to distinguish the current pen-up state or pen-down state of writing. Based on a sliding window, time-domain features and frequency-domain features of the IMU data sequence are extracted, the time period when a pen-up event or pen-down event occurs is detected, and the IMU data sequence is divided into a pen-up interval and a pen-down interval. The writing state is updated, and the inference of the stroke trajectory and stroke displacement calculation model is triggered when the pen is lifted.

[0042] A stroke calculation module is used to calculate the stroke trajectory in the pen-down interval and the stroke displacement in the pen-up interval from the IMU data sequence, store the IMU data sequences of the latest three intervals, preprocess the data of each interval when a pen-up event occurs and then input it into a neural network model, output the stroke trajectory or stroke displacement of the corresponding interval, calculate the global position information of the stroke and optimize it.

[0043] A handwriting display module is used to display the handwriting restored by the model in real time. Each time it receives the coordinate sequence of a complete restored stroke from the stroke calculation module, it draws it on the corresponding display interface to achieve the visual effect of stroke-level restoration.

[0044] The technical effects achieved by the present invention are as follows:

[0045] 1. An efficient writing state classification method: The present invention proposes a pen event detection method based on machine learning for identifying the pen-down or pen-up state occurring from the pen body IMU data generated during the writing process. This method maintains a sliding window, extracts time-frequency domain features from the IMU data sequence for classification, can accurately determine whether a pen-up or pen-down event occurs in the current window, and updates the writing state in real time according to the result. This method has high recognition accuracy and low calculation latency and can adapt to different IMU data frequencies.

[0046] 2. A high-precision handwriting restoration model: The present invention designs a handwriting restoration model based on stroke spatio-temporal context for restoring the handwriting content written from the pen body IMU data generated during the writing process. This model combines the IMU data sequences of adjacent pen-down and pen-up intervals, extracts the spatio-temporal features of the data and jointly calculates the pen-down strokes and pen-up displacements of the corresponding intervals, and calculates the global position of the current stroke according to the global position of the historical strokes. This model has high restoration accuracy and versatility and can adapt to changes in different styluses, IMU installation positions, writing directions, and writing styles.

[0047] 3. Optimize the continuous learning ability of the model: When writing on a device with real reporting point information, the present invention can utilize the handwriting and corresponding IMU information during writing, and use this information for incremental training and personalized fine-tuning of the model to improve the accuracy and robustness of the model. The data-driven optimization mechanism enables the model to continuously optimize and improve under different scenarios and conditions, thereby enhancing the long-term effectiveness of the model.

[0048] 4. Wide application scenarios and good interaction experience: Since the present invention does not require the use of a special stylus, but fixes the IMU sensor on the pen body for use, to a large extent, the original writing experience is retained. The recovery process is automatic without user intervention, and a smooth recovery effect can be achieved, which has good practical promotion value in fields such as education and office. Description of the Drawings

[0049] Figure 1 It is a schematic diagram of the module structure of the handwriting recovery system.

[0050] Figure 2 It is a schematic diagram of an embodiment of the electronic stylus device.

[0051] Figure 3 It is a schematic diagram of the relationship between the IMU body system and the ground system.

[0052] Figure 4 It is a schematic diagram of pen event detection and data interval division.

[0053] Figure 5 It is a schematic diagram of the structure of the stroke recovery model.

[0054] Figure 6 It is a schematic diagram of the combination of strokes and displacements into handwriting. Detailed Implementation Manner

[0055] To make the technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and understandable, the following will be described in detail in conjunction with the drawings.

[0056] The embodiment of the present invention specifically discloses an electronic stylus device and a handwriting recovery method based on an inertial sensor. The device forms a modified electronic pen by installing an IMU on the handwriting pen. The method detects the pen-down and pen-up events through the IMU data during the writing process, and then identifies the writing state and divides the data interval. Then, the strokes and pen-up displacements of the corresponding interval are restored through a model based on spatio-temporal context, and finally, handwriting information is formed. The association of each module of the system is as Figure 1 shown. It mainly includes the following contents:

[0057] 1. Handwriting pen device modified based on IMU

[0058] The stylus device based on the IMU modification has an IMU sensor and a stylus. The IMU sensor is fixed on the stylus, and there is no relative movement between the two during use. The pen body is made of a hard material and does not deform.

[0059] The IMU in this device can use different data channels and different models of IMUs such as 6-axis and 9-axis that can output acceleration and angular velocity. The stylus in this device can use a signature pen, a capacitive pen, etc. for different writing media such as paper writing and electronic screen writing.

[0060] In this embodiment, the IMU is fixed near the pen tail on the side of the pen body, which can reduce the interference to the pen-holding posture and facilitate the connection with the host computer. The IMU is wired to the host computer through a data cable, which can reduce the latency and packet loss rate of data communication. The IMU data is transmitted to the host computer for calculation, which can improve the calculation accuracy and efficiency of the data, as Figure 2 shown.

[0061] In some embodiments, the data output by the IMU includes magnetometer readings (such as a 9-axis IMU), and the pen body includes a magnetic part (such as a magnet or a small motor). The magnetic part of the pen body will interfere with the magnetometer readings of the IMU, reducing the reliability of the output data. In this case, the IMU needs to be fixed on the pen body and then the magnetometer calibration operation is performed to obtain more accurate magnetometer readings. The magnetometer calibration method can use the ellipsoid fitting algorithm, and the process includes rotating the stylus device above the writing plane around each axis to collect the magnetometer data of the IMU in various postures, calculating the offsets of the data on each axis and using them to correct the original readings of the magnetometer.

[0062] 2. Writing state classification based on IMU data

[0063] The writing state classification method based on IMU data classifies the writing state into two categories: the pen-lifting state and the pen-down state. The current writing state is updated by detecting the pen-lifting event and the pen-down event, and the IMU data sequence is divided into intervals. The specific process is as follows:

[0064] 1) Data preparation: The data output by the IMU sensor includes three-axis acceleration and three-axis angular velocity. Some sensors may also provide data such as three-axis magnetometer readings. First, use an attitude estimation method (such as Kalman filtering, Mahony filtering, etc.) to estimate the IMU attitude corresponding to each data frame, and represent it by the quaternion vector Q sensor =[q w ,q x ,q y ,q z , where q w represents half of the cosine of the rotation angle, and q x ,q y,q z Represents half of the product of the rotation angle sine and the axis directions of the rotation axes. Since the original data output by the IMU is based on the IMU body system, and the IMU body system rotates relative to the ground system, the quaternion of each frame is used to rotate the data of that frame to obtain the readings of each data in the ground system, as Figure 3 shown.

[0065] Taking acceleration as an example, let the original acceleration data be A = [a x ,a y ,a z . Through quaternion multiplication, it can be calculated that where V g = [0,a xg ,a yg ,a zg and A g = [a xg ,a yg ,a zg are the acceleration data in the ground system, Q sensor represents the attitude of the IMU sensor, V b represents the quaternion extended from A, is the inverse of Q sensor , V b = [0,a x ,a y ,a z , is quaternion multiplication, and the calculation method is:[[]]

[0066]

[0067] where q = q0 + q1i + q2j + q3k and p = p0 + p1i + p2j + p3k are two quaternions. After rotating the acceleration data to the ground system, the local gravitational acceleration needs to be subtracted from the z-axis. For the data sequence after coordinate rotation, a detection window is set. The window size should not be too long to avoid excessive delay, nor too small to affect the significance of data features, and is used to save the latest data to support subsequent pen event detection.

[0068] 2) Feature extraction: Extract the time-frequency domain features of each channel from the data sequence in the detection window. The available time domain features include mean, maximum, minimum, variance, median, kurtosis, skewness, etc., and the available frequency domain features include energy, time center value of short-time Fourier transform, etc. For the extracted frequency domain features, the time domain features can be extracted again to reduce the dimension of the feature vector. Finally, all features are concatenated into a one-dimensional feature vector for pen event detection.

[0069] 3) Pen event detection: The feature vectors extracted from the detection window are input into a machine learning classification model, and the output classification result represents the pen event detection result. Commonly used classification models include SVM (Support Vector Machine), random forest, etc. The model can be a binary classification model or a ternary classification model. If a binary classification model is used, first judge the current writing state: when in the pen-up state, call the pen-up detection model, and the output result is that a pen-up occurs or no event occurs; when in the pen-down state, call the pen-down detection model, and the output result is that a pen-down occurs or no event occurs. If a ternary classification model is used, the output result is that a pen-up occurs, a pen-down occurs, or no event occurs. According to the result output by the model, if a pen-up or pen-down event occurs, update the corresponding writing state, otherwise keep the original writing state. After the writing state is updated, select a data frame from the window (such as the frame with the largest energy peak) as the frame corresponding to the event occurrence, and divide the data interval accordingly: the interval from the pen-up frame to the pen-down frame is the pen-up interval, and the interval from the pen-down frame to the pen-up frame is the pen-down interval. Each data interval contains the IMU axis data and timestamp sequence collected during that time period, such as Figure 4 as shown

[0070] 3. Handwriting Recovery Model Based on IMU Data

[0071] The handwriting recovery model based on IMU data recovers the pen-up displacement and relative pen-down strokes in the corresponding intervals from the IMU data in the pen-up interval and the pen-down interval. The specific process is as follows:

[0072] 1) Data preprocessing: First, obtain the quaternion vector Q surface , which is used to represent the coordinate system of the writing plane. Q surface can be preset or measured using an IMU that is stationary and aligned with the writing plane coordinate system. The definition of the writing plane coordinate system is: the x-axis points directly in front of the writing, the y-axis points to the right of the writing, and the z-axis points vertically downward to the writing plane. Since the actual writing direction of the user is uncertain, it is necessary to rotate the IMU data to the writing plane coordinate system through Q surface to obtain accurate handwriting in terms of direction. For the data in each interval, use Q surface to rotate it to the writing plane coordinate system. Taking acceleration as an example, calculate where V s = [0, a xs , a ys , a zs and A s = [a xs , a ys , a zsThe acceleration data in the writing plane coordinate system is obtained. Then, the data is normalized using the actual range of each channel, and a time channel is added to the data sequence to represent the timestamp offset of each data frame relative to the first frame of the interval. The data in each interval is resampled to a fixed length (e.g., 200 frames) using resampling methods such as, but not limited to, linear interpolation and spline interpolation. For data intervals whose actual content has not been determined (when no new pen event is detected, the number of data frames in the last data interval keeps increasing), resampling is performed after a complete interval is collected.

[0073] 2) Stroke and displacement inference: Set up three data buffers to retain the data in the latest three data intervals. The latest data frame is inserted into the third data interval. When the reception of the third data interval is completed, the data in each buffer is shifted left to the previous buffer, and the data in the first buffer is discarded. When the system is first started, the first two buffers are filled with zero values. After the pen is lifted, the data in the three intervals that have completed data preprocessing is combined into the input data of the model. At this time, the data in the three buffers comes from consecutive pen-down, pen-up, and pen-down intervals, corresponding to the second-to-last stroke, pen-up displacement, and the last stroke at present. Let the corresponding data be s1, h, s2, then the input data is x = [s1, h, s2]. The handwriting restoration model can be expressed as y = Model(x), where y = [p1, t, p2], p1 and p2 correspond to the relative pen-down strokes (stroke trajectories in the pen-down intervals) of the two pen-down intervals, and t corresponds to the pen-up displacement (stroke displacement in the pen-up interval) of the pen-up interval. The structural schematic of the model is as Figure 5 shown.

[0074] The encoder is used to extract the temporal features of the data. The encoder can be, but not limited to, a recurrent convolutional neural network such as ConvLstm (Reference: Shi X, Chen Z, Wang H, et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting[J]. Advances in neural information processing systems, 2015, 28.). The encoder can be expressed as: e = Encoder(x). Since the pen tip always remains on the two-dimensional writing plane in the pen-down state, while the pen tip can move in the three-dimensional space in the air in the pen-up state, and the motion patterns and features are different. To improve the model's ability to process data in the pen-down interval and the pen-up interval differently, encoders with different weights can be used for the pen-down interval and the pen-up interval.

[0075] Use a decoder to calculate the strokes in the corresponding interval from the temporal features of the data. The decoder can use, but is not limited to, a convolutional neural network. The decoder can be expressed as: d = Decoder(e). As mentioned above, decoders with different weights can be used for the pen-down interval and the pen-up interval. The output result of the decoder is d = [d s1 , d h , d s2 , corresponding to the stroke information of each interval. The output for each interval is a 1×2n-dimensional vector of fixed length. After reshaping it into a 2×n-dimensional vector, it can represent the relative coordinate sequence of the two-dimensional stroke, that is, stroke = [(x0, y0), (x1, y1),..., (x n , y n )], where the value of (x0, y0) is approximately equal to (0, 0), and the value of each (x i , y i ) point represents the coordinate of this point relative to the position of (0, 0). For the pen-down interval, stroke is the relative stroke trajectory of this interval. For the pen-up interval, stroke can be understood as the relative trajectory of the trajectory of the pen tip moving in the air mapped onto the writing plane.

[0076] Since the movement of the pen tip in the pen-up interval is relatively free, it is difficult to directly recover the precise pen-up displacement from the corresponding interval of the IMU data. This method is based on the stroke context information and extracts spatial features to further enhance the recovery accuracy of the pen-up displacement. For the stroke information of the pen-down interval output by the decoder, use a stroke encoder to extract spatio-temporal features from it. The stroke encoder can use, but is not limited to, convolutional and recurrent neural networks. The stroke encoder can be expressed as c = Encoder stroke ([d s1 , d s2 ), where c = [c1, c2] represents the spatio-temporal features of two strokes. Combine the stroke context information to form a joint feature e s = [c1, d s1 , d h , d s2 , c2], and use a displacement decoder to calculate the pen-up displacement from the joint feature. The displacement decoder can use, but is not limited to, a graph convolutional neural network such as GCN (Reference: Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[J]. arXiv preprint arXiv:1609.02907, 2016.). The displacement decoder can be expressed as t = Decoder transition (e s)。The final output of the model is the relative stroke of the two pen-down intervals and the displacement of the pen-up interval.

[0077] 4. Pen Event-Driven Handwriting Recovery System

[0078] Detect pen-up and pen-down events based on IMU data, divide the data intervals, update the user's writing state in real time, send the corresponding data into the handwriting recovery model, combine the pen-down strokes and pen-up displacements into a complete handwriting, and realize the real-time recovery of handwriting. The specific process is as follows:

[0079] 1) Writing State Conversion Mechanism: Set up a state machine, and trigger the state conversion through the input of events. The input events include pen-up events and pen-down events, and the writing states include pen-up state and pen-down state. The initial state is the pen-up state. The action mechanism of the state machine is:

[0080] (1) In the pen-up state: When a pen-up event is input, the state machine does not process it; when a pen-down event is input, the state is converted to the pen-down state, and then the data buffer is moved.

[0081] (2) In the pen-down state: When a pen-down event is input, the state machine does not process it; when a pen-up event is input, the state is converted to the pen-up state, the model is called to infer the stroke and displacement, and then the data buffer is moved.

[0082] 2) Global Handwriting Generation: Combine the relative strokes and displacements output by the handwriting recovery model each time to form a complete writing handwriting. Preset an initial position position=(p0,p1), such as (0,0), as the coordinates of the starting point of the first stroke on the writing plane. When the model outputs the relative stroke and displacement, let the displacement be transition=(d x ,d y ), and the subsequent relative stroke be stroke=[(x0,y0),(x1,y1),…,(x n ,y n ), where n is the number of points included in the stroke. Then if the current is the first stroke to be recovered, move stroke to the position of position, that is, stroke=[(x0+p0,y0+p1),(x1+p0,y1+p1),…,(x n +p0,y n +p1)], and the coordinates of each point represent its coordinates on the writing plane. Then update position: (p0,p1)←(x n +p0,y n +p1). If the current recovered stroke is not the first stroke, first add transition to position, and then move stroke to the position of position, that is, stroke=[(x0+dx +p0, y0 + d y +p1), (x1 + d x +p0, y1 + d y +p1), …, (x n +d x +p0, y n +d y +p1)], finally update position: (p0, p1) ← (x n +d x +p0, y n +d y +p1). By this method, the global position of each stroke on the writing plane can be obtained, that is, a complete handwriting can be obtained, as Figure 6 shown.

[0083] 3) Handwriting optimization: Since the number of points of the stroke output by the model is fixed, it may cause the stroke to have serrations or burrs, etc., affecting the visual effect. Therefore, the stroke can be beautified to make the handwriting more beautiful, but it is not necessary. The beautification of the stroke includes the following steps:

[0084] (1) Stroke resampling: For shorter strokes, too many stroke points are likely to cause burrs on the stroke and may also reduce the rendering efficiency of the stroke; for longer strokes, too few stroke points may cause the stroke to appear serrated. Therefore, according to the actual writing time of the stroke, that is, the time from pen-down to pen-up, the points in the stroke are resampled at a certain sampling rate to balance the time interval between points. Resampling methods can include but are not limited to linear interpolation, spline interpolation, etc.

[0085] (2) Stroke smoothing: Smooth the stroke to make the appearance of the stroke more natural. Smoothing does not change the number of points of the stroke, but only modifies the coordinates of the stroke points. Smoothing methods can include but are not limited to Gaussian filtering, median filtering, etc.

[0086] It should be understood that the methods and systems disclosed in the above embodiments provided by the present invention can be implemented in other ways. For example, the above module division can have other division methods in actual implementation, and multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Each module in the present invention can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium, including several instructions for causing a computer device to execute part or all of the steps of the method described in the present invention. For example, an embodiment of the present invention provides a computer device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing each step in the method of the present invention. For example, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, a disk, an optical disc, etc.), and the computer-readable storage medium stores a computer program. When the computer program is executed by a computer, each step of the method of the present invention is implemented.

[0087] Although the present invention has been disclosed above in embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent replacements made by those of ordinary skill in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention. The protection scope of the present invention shall be defined by the claims.

Claims

1. An electronic stylus device based on an inertial sensor, characterized in that, It includes a stylus and an IMU sensor, and the IMU sensor is fixed on the body of the stylus.

2. The device according to claim 1, wherein The outer shell of the stylus is made of a hard material, and the body is a non-deformable structure; the stylus and the IMU sensor are rigidly connected, and there will be no relative movement between them during use.

3. The device according to claim 1, characterized in that, The installation direction of the IMU sensor is set as follows: the x-axis of the IMU sensor is parallel to the body of the pen and points to the tip direction, and the y-axis and z-axis of the IMU sensor are perpendicular to the body of the pen.

4. The device according to claim 1, characterized in that, The IMU sensor is fixed on the side of the body of the stylus.

5. The device according to claim 1, characterized in that, The IMU sensor sends the acquired data to the host computer for processing, and the IMU sensor communicates with the host computer in a wired or wireless manner.

6. A handwriting restoration method based on an inertial sensor, characterized in that, It includes the following steps: Adopt the electronic stylus device based on inertial sensors described in any one of claims 1 to 5, and collect data frames of the IMU sensor in real time and generate a data sequence; Extract time-domain features and frequency-domain features according to the generated data sequence, detect pen events according to the extracted time-domain features and frequency-domain features, identify the writing state and divide data intervals, and the data intervals include a pen-lift interval and a pen-down interval; Calculate the stroke trajectory in the pen-down interval and the stroke displacement in the pen-lift interval; Perform handwriting restoration according to the stroke trajectory in the pen-down interval and the stroke displacement in the pen-lift interval.

7. The method according to claim 6, wherein The step of extracting time-domain features and frequency-domain features according to the generated data sequence, detecting pen events according to the extracted time-domain features and frequency-domain features, and then identifying the writing state and dividing data intervals includes: Filter the data of each channel of the IMU sensor, perform fusion calculation on acceleration, angular velocity and other available data, and obtain the real-time attitude information of the IMU sensor; Using the obtained attitude information, rotate the data of each channel from the sensor body coordinate system to the ground coordinate system and eliminate the influence of gravity; Set a sliding window with a fixed length, record the latest data generated through the window, and extract the time-domain features and frequency-domain features of the data in the window; Use a pen event classifier based on a machine learning model to classify the time-domain features and frequency-domain features of the data in the current window to detect pen events, and then identify the writing state and divide data intervals; the pen events include a pen-lift event and a pen-down event, and the writing states include a pen-lift state and a pen-down state.

8. The method according to claim 7, characterized in that The time-domain features include the sum, average value, maximum value, minimum value, variance, median, peak value, skewness of the data values of each channel; the frequency-domain features include the peak value, skewness, difference between the maximum value and the minimum value of the energy of the data values of each channel, and the peak value, skewness, difference between the maximum value and the minimum value of the short-time Fourier transform value.

9. The method according to claim 7, characterized in that, The step of calculating the stroke trajectory in the pen-down interval and the stroke displacement in the pen-lift interval includes: Rotate the data of each channel from the ground coordinate system to the writing surface coordinate system according to the predefined writing direction; Set three data buffers to record the data in the latest three data intervals generated; When a pen-lift event is recognized, normalize and resample the data in the three buffers; Input the normalized and resampled data sequence into the encoder, extract the temporal features in the data through spatio-temporal modeling, and generate the hidden layer feature vector; Input the hidden layer feature vector into the decoder to calculate the stroke trajectory in the pen-down interval; Input the stroke trajectory in the pen-down interval into the stroke encoder to extract the stroke features in the data through spatio-temporal modeling; Input the stroke features in the pen-down interval and the stroke trajectories in the three intervals into the displacement decoder to calculate the stroke displacement in the pen-up interval.

10. A handwriting restoration system based on an inertial sensor, characterized in that, It includes: The IMU sensor module is used to collect the data generated by the IMU in real time during writing and generate a data sequence; The writing state classification module is used to extract the time domain features and frequency domain features according to the generated data sequence, detect the pen event according to the extracted time domain features and frequency domain features, and identify the writing state and divide the data interval, where the data interval includes the pen-up interval and the pen-down interval; The stroke calculation module is used to calculate the stroke trajectory in the pen-down interval and the stroke displacement in the pen-up interval; The handwriting display module is used to restore the handwriting according to the stroke trajectory in the pen-down interval and the stroke displacement in the pen-up interval and draw it on the display interface.

Images