Generating writing based on movements of stylus

Abstract

A method performed by a computing device includes receiving, from an electronic stylus, a series of three-dimensional accelerometer measurements of movement of the electronic stylus; determining, based on the series of three-dimensional accelerometer measurements, a symbol being written; and generating two-dimensional writing based on the series of three-dimensional accelerometer measurements and the determined symbol.

Description

GENERATING WRITING BASED ON MOVEMENTS OF STYLUSTECHNICAL FIELD

[0001] This description relates to receiving and interpreting input for writing.BACKGROUND

[0002] Users may desire to write with an electronic stylus. However, capacitive surfaces may not always be available to detect movement by the electronic stylus.SUMMARY

[0003] A computing device generates writing based on movement of an electronic stylus. The computing device receives a series three-dimensional accelerometer measurements from the electronic stylus. Based on the accelerometer measurements, the computing device determines a symbol that is being written. The computing device generates two-dimensional writing based on the accelerometer measurements and the determined symbol. No capacitive surface is needed. A user can, for example, “write” with the electronic stylus onto a surface such as a desk or notepad. The writing can be presented within an augmented reality environment.

[0004] According to an example, a method performed by a computing device includes receiving, from an electronic stylus, a series of three-dimensional accelerometer measurements of movement of the electronic stylus; determining, based on the series of three-dimensional accelerometer measurements, a symbol being written; and generating two-dimensional writing based on the series of three-dimensional accelerometer measurements and the determined symbol.

[0005] According to an example, a non-transitory computer-readable storage medium comprises instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device to receive, from an electronic stylus, a first series of accelerometer measurements; present initial writing based on the first series of accelerometer measurements; receive a second series of accelerometer measurements; determine a symbol based on the first series of accelerometer measurements and the second series of accelerometer measurements;modify the initial writing based on determining the symbol; and present additional writing based on the symbol and the second series of accelerometer measurements.

[0006] According to an example, a computing device includes at least one processor and a non- transitory computer-readable storage medium. The non-transitory computer-readable storage medium comprises instructions stored thereon that, when executed by the at least one processor, cause the computing device to receive, from an electronic stylus during a first time period, a first series of accelerometer measurements; determine a first symbol based on the first series of accelerometer measurements; generate first writing based on the first series of accelerometer measurements and the first symbol; receive, from the electronic stylus during a second time period, a second series of accelerometer measurements, the second time period being after the first time period; determine a second symbol based on the second series of accelerometer measurements, the second symbol being different than the first symbol; and generate second writing based on the second series of accelerometer measurements and the second symbol.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 shows a user at a desk wearing a head-mounted device and writing with an electronic stylus.

[0009] FIG. 2 shows movement by the electronic stylus.

[0010] FIG. 3 is a graph showing measurements of movements by the electronic stylus and letters written while the movements are measured.

[0011] FIG. 4 shows an architecture of a transformer generating writing based on the measurements of the movements by the electronic stylus.

[0012] FIG. 5A shows the head-mounted device and a scene, including the electronic stylus and writing, seen by the user who is wearing the head-mounted device.

[0013] FIG. 5B shows a computing device presenting writing based on measured movements of the electronic stylus.

[0014] FIG. 6 shows the electronic stylus moving along a capacitive surface to train a computing device to determine writing based on measured movements of the electronic stylus.

[0015] FIG. 7 is a graph showing a confidence threshold for changing previously-presented writing as a function of time since the writing was presented.

[0016] FIG. 8 is a block diagram of a computing device for generating writing based on movements of an electronic stylus.

[0017] FIGs. 9 A, 9B, and 9C show an example of the head-mounted device.

[0018] FIG. 10 is a flowchart showing a method performed by a computing device according to an example.

[0019] FIG. 11 is a flowchart showing a method performed by a computing device according to an example.

[0020] FIG. 12 is a flowchart showing a method performed by a computing device according to an example.

[0021] Like reference numbers refer to like elements.DETAILED DESCRIPTION

[0022] A user can write with an electronic stylus on a surface, such as a table, a notepad, or a clipboard. The surface does not need to be capacitive, electronic, or have any means of detecting movement of the electronic stylus. A computing device generates two-dimensional writing based on measured three-dimensional movements, such as multiple three-dimensional accelerometer measurements, of the electronic stylus. The measured three-dimensional movements can be received by the computing device in a series that is ordered based on times of measurement. The electronic stylus includes an accelerometer that measures and transmits measurements, such as three-dimensional accelerometer measurements, to the computing device.

[0023] In some examples, the user makes writing motions with the stylus without the stylus contacting a surface, such as by writing into the air. In some examples, the user writes with the stylus alternating between contacting a surface and not contacting a surface. The computing device can generate two-dimensional writing based on measured three-dimensional movements with or without contact by the electronic stylus with a surface.

[0024] The computing device can include a head-mounted device (such as smartglasses or goggles that create an augmented reality environment), a laptop or desktop computer, or a smartwatch. A technical problem with generating two-dimensional writing based on measured three-dimensional movements is that noise or jerky or irregular movements of the electronicstylus can cause undesired marks within the two-dimensional writing. For example, a portion of the stylus that includes an accelerometer and / or gyroscope (which measures the three- dimensional movements) can move while an end portion of the stylus is stationary with respect to the surface, or movement of the portion of the stylus that includes an accelerometer and / or gyroscope can be disproportionate compared to movement of the end portion of the stylus with respect to the surface. A technical solution to this technical problem is for the computing device to determine a symbol that is being written and correct, modify, or augment the determination of the two-dimensional writing based on the determined symbol.

[0025] The computing device determines a symbol, such as a letter or a shape, that the user is writing. The computing device can determine the symbol based on the measurements of the user drawing or writing a portion of the symbol, based on a prediction of the current symbol that is based on one or more previous symbols (such as based on a Markov chain), or combinations thereof.

[0026] Based on the determined symbol, the computing device can modify portions, such as strokes, of two-dimensional writing that are included in the symbol. The computing device can, for example, smooth out, or remove, some of the measured movements based on strokes that make up the determined symbol. A technical benefit of modifying portions of the two- dimensional writing based on the determined symbol is that the computing device can thereby present the writing in a style that the user wrote the writing in, rather than in a predetermined font, without some of the jerks measured by the electronic stylus.

[0027] FIG. 1 shows a user 102 at a desk 112 wearing a head -mounted device 104 and writing with an electronic stylus 108. The head-mounted device 104 is an example of a computing device. The head-mounted device 104 can include, for example, augmented reality glasses or virtual reality goggles. The head-mounted device 104 is mounted on a head 106 of the user 102. The user 102 holds the electronic stylus 108 in a hand 110 of the user 102. The user 102 writes on the desk 112 with the electronic stylus 108 as if the electronic stylus 108 were a writing utensil such as a pen or a pencil. In the example of FIG. 1, the user 102 is sitting on a chair 114.

[0028] A computing device, such as the head-mounted device 104, generates and / or presents virtual writing 116 based on the movement of the electronic stylus 108 by the user 102. The virtual writing 116 can be writing presented in a display of the computing device such as the head -mounted device 104. The virtual writing 116 corresponds to the movements of theelectronic stylus 108 by the user 102 The virtual writing 116 can be presented within an augmented reality environment created by the computing device (such as the head-mounted device 104), and thereby appear to be written on the desk 112 as if the user 102 was writing with a pen or a pencil. In the example shown in FIG. 1, the virtual writing 116 includes a portion of the word, “writing,” the portion including the letters, ‘w’, ‘r’, ‘i’, and, ‘t’.

[0029] The electronic stylus 108 includes at least one movement sensor. The movement sensor can include an accelerometer and / or a gyroscope. The accelerometer measures acceleration of a portion of the electronic stylus 108, such as an end portion of the electronic stylus 108. The gyroscope measures rotation and / or angular velocity of the electronic stylus 108 with an axis oriented along a longitudinal axis of the electronic stylus 108.

[0030] The electronic stylus 108 can send multiple accelerometer measurements to a computing device, such as the head-mounted device 104. The electronic stylus 108 can send the multiple accelerometer measurements to the computing device in a series, such as a time-ordered series of accelerometer measurements. The series of accelerometer measurements can be ordered based on times of measurement. In some examples, measurements can be three-dimensional accelerometer measurements measured by the electronic stylus 108 while the electronic stylus 108 was moving. In some examples, measurements can be three-dimensional accelerometer measurements measured by the electronic stylus 108 during movement of the electronic stylus 108. In some examples, the computing device can determine, based on the multiple accelerometer measurements, a symbol that the user 102 is writing with the electronic stylus 108. In some examples, the computing device can determine, based on the multiple accelerometer measurements, a symbol being written with the electronic stylus 108. Based on the determined symbol and the series of accelerometer measurements, the computing device can generate and / or present two-dimensional writing, such as the virtual writing 116.

[0031] In some examples, the computing device (such as the head-mounted device 104) includes a camera (not shown in FIG. 1). The camera can capture images of the electronic stylus 108 and other objects in the scene, such as the desk 112 or other surface on which the user 102 writes with the electronic stylus 108. Based on at least one image captured by the camera, the computing device (such as the head-mounted device 104) can determine an initial location 118 of the electronic stylus 108. The initial location 118 can be a location on the surface, such as the surface of the desk 112, at which the user 102 began writing with the electronic stylus 108. Thecomputing device can generate and / or present the virtual writing 116 based on the initial location 118. The computing device can generate and / or present a beginning of a first stroke of the virtual writing 116 at the initial location, and displacements, which can be based on the accelerometer measurements, can be measured, generated, and / or displayed from the initial location 118.

[0032] In some examples, the electronic stylus 108 includes an input sensor such as a button. In some examples, the electronic stylus 108 prompts the computing device to generate writing based on measurements in response to the user 102 activating the input sensor, such as the user 102 pressing the button.

[0033] FIG. 2 shows movement 204 by the electronic stylus 108. The electronic stylus 108 is moving along a surface 202. In some examples, the surface 202 is a flat surface. The surface 202 can include, for example, a surface of a desk, a surface of a notepad, or a surface of a clipboard.

[0034] The accelerometer and / or gyroscope of the electronic stylus 108 measure the shown movement 204 of the electronic stylus 108. The accelerometer can measure the movement 204 as three-axis accelerometer measurements 206. The three-axis accelerometer measurements 206 can be measured in three orthogonal directions and / or axes as a function of time. The three-axis accelerometer measurements 206 can form a series. The series can include three-tuples, with each tuple including a measurement in one of the three orthogonal directions. The series of three- axis accelerometer measurements can be a time series. The series of three-axis accelerometer measurements can be ordered based on times of measurement.

[0035] The gyroscope can measure the movement 204 as three-axis gyroscope measurements 208. In some implementations, the gyroscope measurements 208 are measured about only one axis that extends along a longitudinal axis of the electronic stylus 108. In some implementations, the three-axis gyroscope measurements 208 are measured with freedom of rotation in all three axes. The gyroscope measurements can form a series. The series can include one-tuples with gyroscope measurements measured about only one axis, or three-tuples with gyroscope measurements measured about three axes. The series of gyroscope measurements can be a time series. The series of gyroscope measurements can be ordered based on times of measurement.

[0036] The user 102 (not shown in FIG. 2) can write symbols with the electronic stylus 108. The user 102 can write the symbols with the electronic stylus 108 by sliding and / or moving an end portion 210 of the electronic stylus 108 along the surface 202. In some examples, the symbols can include alphanumeric characters, such as letters and / or numbers within an alphabet. In someexamples, the symbols can include geometric shapes, such as polygons (including rectangles or triangles) or ovals.

[0037] In some examples, the symbols can be composed of one or multiple strokes. A stroke can include, for example, a line or curve, or a circular or semi-circular element. In some examples, a stroke can be formed from continuous movement of the end portion 210 along the surface 202. In some examples, a stroke includes movement of the end portion 210 along the surface 202 without a velocity of the end portion 210 becoming zero. In some examples, a stroke includes movement of the end portion 210 without the end portion 210 moving off of the surface 202. In some examples, a stroke includes movement of the end portion 210 before and / or until the end portion 210 moves off of the surface 202. Movement of the end portion 210 off of the surface 202 can be determined based on measured acceleration and / or determined velocity in a z-axis and / or an axis orthogonal to the surface 202.

[0038] In some examples, the computing device generates two-dimensional writing based on determining multiple strokes. The computing device determines the multiple strokes based on the series of three-axis accelerometer measurements 206. The computing device can separate the accelerometer measurements into different sets or series, with each set or series corresponding to a different stroke. The computing device can, for example, determine a first stroke based on a first set or series of multiple accelerometer measurements and the determined symbol, and determine a second stroke based on a second set or series of accelerometer measurements and the determined symbol. The number of sets or series of accelerometer measurements can correspond to the number of strokes in the symbol. For example, the letter ‘w’ is made of four strokes, one for each diagonal line; the letter ‘r’ is made of two strokes, the stem formed from a line and the curve; the letter ‘i’ is made of two strokes, one for the line and one for the dot; and the letter ‘f is made of two strokes, one for the vertical stem and one for the horizontal line.

[0039] FIG. 3 is a graph showing measurements of movements by the electronic stylus 108 (not shown in FIG. 3) and letters written while the movements are measured. The graph shows acceleration 302 as a function of time (‘f ) 304. The first and / or top row shows letters 306 spelling “written,” and broken down as, ‘w,’ ‘r,’ ‘i,’ ‘t,’ ‘i,’ ‘n,’ and ‘g.’ The second row shows measured acceleration in the x-axis 308 (to the left and right of the user 102) as a function of time and with respect to a zero value (shown by a broken line). The values in the second row represent a series, or multiple series, of accelerometer measurements in the x-axis 308. The thirdrow shows measured acceleration in the y-axis 310 (toward and away from the user 102 while moving along the desk 112) as a function of time and with respect to a zero value (shown by a broken line). The values in the third row represent a series, or multiple series, of accelerometer measurements in the y-axis 310. The fourth row shows measured acceleration in the z-axis 312 (perpendicular to a plane extending along a surface of the desk 112, for example) as a function of time and with respect to a zero value (shown by a broken line). The values in the fourth row represent a series, or multiple series, of accelerometer measurements in the z-axis 312. While the measurements shown in the second row, third row, and fourth row of FIG. 3 appear continuous, the measurements may represent discrete measurements that form a series (or multiple series) in each of the second row, the third row, and the fourth row. The discrete measurements can represent discrete, time-sampled signals. Three-tuples of accelerometer measurements within a series (or multiple series) can be formed from values of the measured acceleration in the x-axis 308, measured acceleration in the y-axis 310, and measured acceleration in the z-axis 312 with same time values (same values for t 304).

[0040] The combination of acceleration values in the x-axis 308 and acceleration values in the y- axis 310 form, and / or represent, the letters 306. The acceleration values in the x-axis 308 and y- axis 310 are caused by movement of the electronic stylus 108 along a surface, such as the surface of the desk 112. Letters are examples of symbols represented and / or formed by measured acceleration.

[0041] The computing device can recognize symbols, such as letters 306 in the example of FIG. 3, as being written by the electronic stylus 108 during distinct time periods, such as the time periods 314, 316, 318, 320, 322, 324, 326 shown in FIG. 3. In some examples, the computing device determines the time periods based on the acceleration values in the x-axis 308 and the acceleration values in the y-axis 310 corresponding to and / or matching predetermined symbols. In some examples, a series of accelerometer values includes accelerometer values within a distinct time period, such as the time periods 314, 316, 318, 320, 322, 324, 326, corresponding to a symbol. In some examples, a series of accelerometer values includes accelerometer values corresponding to a stroke within a symbol.

[0042] In some examples, the computing device determines the time periods based on the acceleration values in the z-axis 312. The acceleration values in the z-axis can represent the user 102 lifting the electronic stylus 108 between symbols, or between strokes within symbols. Thelifting of the electronic stylus 108 can be similar to a writer lifting a pen to avoid writing on paper while moving the pen to a different location.

[0043] The computing device can determine temporal boundaries, such as the temporal boundaries 328, 330, 332, 334, 336, 338 shown in the example of FIG. 3, that separate the time periods 314, 316, 318, 320, 322, 324, 326. The computing device can determine the temporal boundaries based on the accelerometer measurements in the z-axis 312, which indicates accelerometer measurements in a direction orthogonal to a two-dimensional plane (represented by the x-axis 308 and the y-axis 310) in which a majority of movements are measured by accelerometer measurements. In some examples, the two-dimensional plane represented by the x-axis 308 and the y-axis 310 is determined by a majority of movements other than during times when movements and / or accelerometer measurements are detected and / or measured in the orthogonal direction and / or axis (the z-axis). In some examples, movements and / or accelerometer measurements are detected and / or measured in the orthogonal direction and / or axis (the z-axis) less frequently than measurements in the other two directions and / or axes (the x- axis 308 and the y-axis 310). The accelerometer measurements with respect to the x-axis 308 and the y-axis 310 can be a majority of the accelerometer measurements because the user 102 is moving the electronic stylus 108 along the surface, such as the surface of the desk 112, and moving the electronic stylus 108 away from or toward the surface less frequently (such as between letters and between strokes within the two instances of ‘i’ and the ‘f ) than along the surface. In some examples, the computing device determines the temporal boundaries at peak values of the accelerometer measurements with respect to the z-axis 312. In some examples, the computing device determines the temporal boundaries at zero values following peak values of the accelerometer measurements with respect to the z-axis 312, which represent inflection points when the velocity is zero during times at which the user 102 is moving the electronic stylus 108 back toward the surface.

[0044] In some examples, the computing device divides the accelerometer measurements into sets or series. The sets or series can each be based on a symbol that is written, a time period 314, 316, 318, 320, 322, 324, 326 during which the user 102 is writing, and / or based on temporal boundaries 328, 330, 332, 334, 336, 338. For example, the computing device can determine that a first symbol, ‘w,’ is being written during first time period 314 based on a first set or series of accelerometer measurements that are measured during first time period 314; that a secondsymbol, ‘r,’ is being written during second time period 316 based on a second set or series of accelerometer measurements that are measured during second time period 316; that a third symbol, ‘i,’ is being written during third time period 318 based on a third set or series of accelerometer measurements that are measured during third time period 318; that a fourth symbol, ‘t,’ is being written during fourth time period 320 based on a fourth set or series of accelerometer measurements that are measured during fourth time period 320; that a fifth symbol, ‘i,’ is being written during fifth time period 322 based on a second set or series of accelerometer measurements that are measured during fifth time period 322; that a sixth symbol, ‘n,’ is being written during sixth time period 324 based on a sixth set or series of accelerometer measurements that are measured during sixth time period 324; and that a seventh symbol, ‘g,’ is being written during second time period 326 based on a seventh set or series of accelerometer measurements that are measured during seventh time period 326.

[0045] In some examples, the computing device determines the time periods 314, 316, 318, 320, 322, 324, 326 and / or sets or series of accelerometer measurements based on the determined temporal boundaries 328, 330, 332, 334, 336, 338. In some examples, the computing device determines the time periods 314, 316, 318, 320, 322, 324, 326 and / or temporal boundaries 328, 330, 332, 334, 336, 338 based on determinations of symbols that are being written, with the determinations of the symbols being based on the accelerometer measurements. In some examples, the computing device determines the time periods 314, 316, 318, 320, 322, 324, 326 and / or temporal boundaries 328, 330, 332, 334, 336, 338 based on determinations that different symbols are being written, such as determining the first boundary 328 and / or first and second time periods 314, 316 based on determining that the first symbol (e.g. ‘w’) that is being written during the first time period 314 is different than the second symbol (e.g. ‘r’) that is being written during the second time period 316.

[0046] In some examples, the computing device determines that the first symbol (e.g. ‘w’) is being written based on the first set or series of accelerometer measurements and generates writing based on the first symbol and the first set or series of accelerometer measurements, determines that the second symbol (e.g. ‘r’) is being written based on the second set or series of accelerometer measurements and generates writing based on the second symbol and the second set or series of accelerometer measurements, determines that the third symbol (e.g. ‘i ’ ) is being written based on the third set or series of accelerometer measurements and generates writingbased on the third symbol and the third set or series of accelerometer measurements, determines that the fourth symbol (e.g. ‘t’) is being written based on the fourth set or series of accelerometer measurements and generates writing based on the fourth symbol and the fourth set or series of accelerometer measurements, determines that the fifth symbol (e.g. ‘i’) is being written based on the fifth set or series of accelerometer measurements and generates writing based on the fifth symbol and the fifth set or series of accelerometer measurements, determines that the sixth symbol (e.g. ‘n’) is being written based on the sixth set or series of accelerometer measurements and generates writing based on the sixth symbol and the sixth set or series of accelerometer measurements, and determines that the seventh symbol (e.g. ‘g’) is being written based on the seventh set or series of accelerometer measurements and generates writing based on the seventh symbol and the seventh set or series of accelerometer measurements.

[0047] FIG. 4 shows an architecture of a transformer generating writing based on the measurements 206, 208 of the movements 204 by the electronic stylus 108. The transformer receives measurements 402. The measurements 402 can include the series of three-axis accelerometer measurements 206 and / or the series of three-axis gyroscope measurements 208. The transformer estimates locations and / or coordinates of an end portion of the electronic stylus 108 based on the three-axis accelerometer measurements 206 and / or three-axis gyroscope measurements 208. The computing device generates the writing, such as the virtual writing 116 shown in FIG. 1, based on the determined movements based on the measurements 206, 208, as well determined locations and / or coordinates such as the initial location 118.

[0048] The architecture of the transformer includes a convolutional backbone 404. The convolutional backbone 404 receives the measurements 402. The convolutional backbone 404 can be a component of a neural network, such as a convolutional neural network, included in the architecture of the transformer. The convolutional backbone 404 can extract features from the measurements 402 to process the measurements 402 into feature representations, such as symbols and / or strokes that are components of the symbols.

[0049] The convolutional backbone 404 can provide the extracted features to a sequence embedding module 406 and / or to a positional embedding module 408. The sequence embedding module 406 and / or positional embedding module 408 can be included in the architecture of the transformer.

[0050] The sequence embedding module 406 can predict and / or determine a current stroke and / or symbol that is being written based on a previous sequence or series and / or based on the previous sequence or series and recent measurement data. In some examples, the sequence embedding module 406 predicts and / or determines the current stroke and / or symbol based on Markov chains. For example, if the previous symbol was the letter, ‘f , then the current symbol is more likely to be ‘h’ than ‘b’.

[0051] The positional embedding module 408 can determine one or more positions of the electronic stylus 108 based on the features received from the convolutional backbone 404. The positional embedding module 408 can, for example, determine locations and / or strokes of the electronic stylus 108 based on the received features.

[0052] In some examples, the architecture of the transformer includes a transformer 410. In some examples, the transformer 410 receives the predicted and / or determined stroke and / or symbol from the sequence embedding module 406. In some examples, the transformer 410 receives the determined locations and / or strokes from the positional embedding module 408. In some examples the transformer 410 generates writing, such as symbols (which can include letters within an alphabet or geometric shapes such as ovals or rectangles) or components of symbols such as strokes, based on the determined symbol and / or the determined positions of the electronic stylus 108. The transformer 410 can, for example, remove movements that stray from predetermined patterns that are included in determined symbols.

[0053] In some examples, the architecture of the transformer includes a regression head 412. The regression head 412 determines whether the writing generated by the transformer 410 should be outputted or discarded. Outputting the generated writing can include presenting the writing on a display of the computing device. Discarding the generated writing can include deleting or otherwise not presenting the generated writing.

[0054] The regression head 412 can determine whether to output or discard the generated writing based on comparing a confidence value of the generated writing to a confidence threshold. The regression head 412 can determine the confidence value based on comparing the generated writing to predetermined symbols and / or components of predetermined symbols such as strokes. The predetermined symbols can be computer-generated symbols such as fonts for letters or geometric shapes, or can be based on measurements of symbols written by trainers or users ofthe computing device. An example of generating measurements of symbols by training is shown and described with respect to FIG. 6.

[0055] FIG. 5 A shows the head-mounted device 104 and a scene, including the electronic stylus 108A, 108B and writing 116A, 116B, as seen by the user 102 who is wearing the head-mounted device 104. In the example of FIG. 5 A, the head-mounted device 104 includes two lenses, a left lens 502A and a right lens 502B. The lenses 502A, 502B are transparent, enabling the user 102 to see objects within the scene beyond the lenses 502A, 502B. The scene viewed by the user 102 is the same as, and / or similar to, the scene shown in FIG. 1.

[0056] The user 102 can see, with a left eye of the user 102 through the left lens 502A, a first representation of the hand 110A of the user 102 holding the stylus 108 A. The hand 110A and electronic stylus 108A shown in FIG. 5A are the actual hand 110 and electronic stylus 108 as seen by the left eye of the user 102 through the left lens 502A. The user 102 can also see, with a right eye of the user 102 through the right lens 502B, a second representation of the hand HOB of the user 102 holding the stylus 108B. The hand 110B and electronic stylus 108B shown in FIG. 5 A are the actual hand 110 and electronic stylus 108 as seen by the right eye of the user 102 through the right lens 502B.

[0057] In the example shown in FIG. 5 A, the computing device and / or head-mounted device 104 augments the image(s) seen by the user 102 by presenting writing to the user 102. The presented writing was generated based on measured movements of the electronic stylus 108. In some examples, the initial location 118 of the presented writing is based on one or more images captured by a camera included in the computing device such as the head-mounted device 104.

[0058] In the example shown in FIG. 5 A, the head-mounted device 104 displays writing 116A on the left lens 502A. The user 102 views the writing 116A with the left eye of the user 102. In the example shown in FIG. 5 A, the head-mounted device 104 displays writing 116B on the right lens 502B. The user 102 views the writing 116B with the right eye of the user 102. The writing 116A, 116B is visible to the user 102, as if the writing were actually on the desk 112 (not shown in FIG. 5A). The writing 116A, 116B can represent the virtual writing 116 described above. The writing 116A, 116B presented by the head-mounted device 104 is based on the writing that the computing device generated based on the measurements 206, 208.

[0059] FIG. 5B shows a computing device 550 presenting writing 556 based on measured movements of the electronic stylus 108. In this example, the user 102 (not labeled in FIG. 5B)holds the electronic stylus 108 in the actual hand 1 10 of the user 102 and makes writing motions with the electronic stylus 108 onto the desk 112, as described above. In this example, a computing device 550, such as a laptop or notebook computer, is in communication with the electronic stylus 108.

[0060] In the example shown in FIG. 5B, the computing device 550 presents writing 556 on a display 552 included in the computing device 550. The computing device 550, and / or another computing device in communication with the computing device 550, generates the writing 556 based on the movements and / or motions of the electronic stylus 108.

[0061] A computing device can be trained to generate writing, such as the writing 116A, 116B, 556 described above, based on movements of a stylus along a surface that can detect the movements of the stylus. In some examples, the computing device is trained based on comparisons of measurements of movements (such as accelerometer measurements and / or measurements by a gyroscope) with detections of a stylus on the surface. In some examples, the surface that detects the movements and / or locations of the stylus is a capacitive surface.

[0062] FIG. 6 shows the electronic stylus 108 moving along a capacitive surface 602 to train a computing device to determine writing based on measured movements of the electronic stylus. In some examples, the electronic stylus 108 includes an accelerometer that measures acceleration of the electronic stylus 108. A series of three-axis accelerometer measurements 606 measured by the accelerometer are shown in FIG. 6. The series of three-axis accelerometer measurements 606 can correspond to the series of three-axis accelerometer measurements 206 shown and described with respect to FIG. 2. In some examples, the electronic stylus 108 includes a gyroscope that measures rotation of the electronic stylus 108. A series of three-axis gyroscope measurements 608 measured by the gyroscope are shown in FIG. 6. The series of three-axis gyroscope measurements 608 can correspond to the three-axis gyroscope measurements 208 shown and described with respect to FIG. 2.

[0063] The capacitive surface 602 can detect actual locations of contact by the electronic stylus 108 onto the capacitive surface 602 during movement 604 of the electronic stylus 108. The capacitive surface 602 can include a trackpad, and / or be constructed in a similar manner to a trackpad. The capacitive surface 602 can detect the actual locations of contact by the electronic stylus 108 onto the capacitive surface 602 based on changes of capacitance of the capacitive surface 602 that are caused by contact by a metal (or otherwise conductive) end portion 210 ofthe electronic stylus 108 with the capacitive surface 602. The computing device, and / or another device that is training a neural network (such as the transformer architecture shown and described with respect to FIG. 4) that will be accessed by the computing device, compares the series of three-axis accelerometer measurements 606 measured by the accelerometer and / or series three-axis gyroscope measurements 608 measured by the gyroscope to the locations of the electronic stylus 108 on the capacitive surface 602 determined by the capacitive surface 602. The trained neural network can thereafter predict locations of the electronic stylus 108 on surfaces that do not detect contact by the electronic stylus 108 based on accelerometer measurements and / or gyroscope measurements. In some examples, the neural network can be trained to determine and / or recognize symbols based on symbols written onto the capacitive surface 602.

[0064] The capacitive surface 602 is an example of determining actual locations of contacts by the end portion 210 onto a surface. Other examples include resistive surfaces or cameras that capture images of the electronic stylus 108 writing onto a surface. Locations of the electronic stylus 108, and / or end portion 210 of the electronic stylus, can be determined based on images captured by a camera. Locations determined based on images captured by a camera can generate data for determining movement of an electronic stylus when the electronic stylus is not in contact with a surface.

[0065] In some examples, the computing device can change previously-displayed writing, such as writing 116A, 116B, 556, based on additional information received after the writing was presented. For example, after a portion of a symbol (such as a stroke that is a component of a letter or shape) has been presented, the computing device can determine a symbol that a user is intending to write. Based on the determined symbol, as well as the series of measured movements of the electronic stylus 108, the computing device can determine a different presentation of the writing than the computing device previously determined based on the measured movements without the determination of the symbol (or based on a previous determination that a different symbol was intended).

[0066] In some examples, the computing device changes the writing, a portion of the writing, and / or a stroke or portion of a stroke, based on the determination of the different presentation. In some examples, the computing device changes the writing, portion of the writing, and / or stroke or portion of the stroke, based on a time that has elapsed since displaying the writing, portion of the writing, and / or stroke or portion of the stroke. The computing device can change the writing,portion of the writing, and / or stroke or portion of the stroke to the determined different presentation based on satisfaction of a time threshold, such as the writing, portion of the writing, and / or stroke or portion of the stroke having been presented within the time threshold of time since the determination that a different presentation of the writing should be displayed.

[0067] In some examples, the computing device changes the writing, portion of the writing, and / or stroke or portion of the stroke, based on a confidence level of the determination of the different presentation, and / or based on a confidence level of a change to the display of the writing, portion of the writing, and / or stroke or portion of the stroke. In some examples, the confidence level is a confidence level that the determination of the different presentation is the correct presentation of the writing, stroke, and / or portion of the stroke based on the series of measured movements. In some examples, the confidence level is a difference and / or increase in the confidence level between the confidence level of the writing, stroke, or portion of the stroke that was presented and the confidence level of the determined different presentation of the writing, stroke, or portion of the stroke. The computing device can change the writing, portion of the writing, and / or stroke or portion of the stroke to the determined different presentation based on satisfaction of a confidence threshold.

[0068] In some examples, the computing device determines whether to change the writing to the determination of the different presentation based on both the time since displaying the writing, portion of the writing, and / or stroke or portion of the stroke, and the confidence level. In some examples, the confidence threshold can be higher for greater times since displaying the writing, portion of the writing, and / or stroke or portion of the stroke.

[0069] FIG. 7 is a graph showing a confidence threshold 706 for changing previously-presented writing as a function of time since the writing was presented. The graph shows time 702 as an independent variable and confidence 704 as a dependent variable. In this example, the computing device determines whether to change the writing to the determination of the different presentation based on both the time since displaying the writing, portion of the writing, and / or stroke or portion of the stroke and the confidence level. In some examples, the confidence threshold is higher for greater times since displaying the writing, portion of the writing, and / or stroke or portion of the stroke. The confidence threshold 706 as a function of time 702 shown in FIG. 7 is merely an example. In some examples, the confidence threshold is asymptotic as a function of time 702, with a maximum confidence threshold that does not continue to increasesignificantly for greater time 702 values. Tn some examples, a maximum time 702 value exists after which the computing device will not change the presentation of the writing, portion of the writing, and / or stroke or portion of the stroke no matter how high the confidence level is.

[0070] FIG. 8 is a block diagram of a computing device 800 for generating writing based on movements of an electronic stylus 108. The computing device 800 can be an example of the head-mounted device 104, the computing device 550, and / or any computing device that receives movement measurements from the electronic stylus 108 and / or generates writing, such as the writing 116A, 116B, 556 based on the movement measurements.

[0071] The computing device 800 includes a measurement processor 802. The measurement processor 802 processes measurements received from the electronic stylus 108. The measurements can include a series of accelerometer measurements, such as three-axis accelerometer measurements, and / or gyroscope measurements, such as three-axis gyroscope measurements 208. The measurement processor 802 can, for example, determine initial accelerations, velocities, and / or locations (with respect to a starting point) based on the measurements received from the electronic stylus 108.

[0072] The computing device 800 can include a symbol determiner 804. The symbol determiner 804 can determine a symbol that is presently being written by the user 102 with the electronic stylus 108. In some examples, the symbol can include a letter within an alphabet. In some examples, the symbol can include an alphanumeric character, such as a letter or number within an alphabet and number system. In some examples, the symbol can include a character such as a letter, a number, or a punctuation mark. In some examples, the symbol can include a geometric shape, such as an oval, a rectangle, or a triangle.

[0073] In some examples, the symbol determiner 804 determines the symbol based on the series of measurements processed by the measurement processor 802. In some examples, the symbol determiner 804 determines the symbol based on the series of measurements processed by the measurement processor 802 and a neural network that has been trained based on measurements of movements and detections of locations on a surface, as described above with respect to FIGs.4 and 6. The symbol determiner 804 can determine the symbol based on the series of measurements caused by movements of the electronic stylus 108 that make up a portion of the symbol. In some examples, the symbol determiner 804 can determine the symbol based onmeasurements caused by one or more strokes of the electronic stylus 108 that make up a portion of the symbol.

[0074] Strokes can be components of letters or other symbols. Strokes can be continuous movements of the electronic stylus 108 along the surface (such as the desk 112) after placing the electronic stylus 108 onto the surface and before lifting the electronic stylus 108 off of the surface. Strokes can be straight, as in the letters, ‘k’, T, ‘v’ , ‘w’, ‘x’, or ‘z’, or can be curved, as in the letters, ‘c’, ‘o’, or ‘s’. Straight strokes can be horizontal, vertical, or diagonal. Curved strokes can be open (not returning to the original location as in ‘c’ or ‘s’) or closed (returning to the original location (as in ‘o’).

[0075] In some examples, the symbol determiner 804 determines the symbol based on previous symbols. The symbol determiner 804 can, for example, predict a current symbol based on previous symbols. The symbol determiner 804 can, for example, use Markov chains or other predictive models to predict the current symbol based on one or more previous symbols.

[0076] In some examples, the symbol determiner 804 determines the current symbol based on a combination of the previous symbols and series of measurements caused by movements and / or strokes that form a portion of the current symbol. The symbol determiner 804 can, for example, determine a most likely current symbol based on the combination of the previous symbols and the series of measurements caused by movements and / or strokes that form the portion of the current symbol. The symbol determiner 804 can, for example, determine a likelihood and / or confidence level that the determined current symbol is the symbol that the user 102 intends to write and / or complete.

[0077] The symbol determiner 804 can include a boundary determiner 806. In some examples, the boundary determiner 806 can determine boundaries between symbols. In some examples the boundary determiner 806 can determine boundaries between measurements that are based on movements of the electronic stylus 108. The boundaries 328, 330, 332, 334, 336, 338 shown and described with respect to FIG. 3 are examples of boundaries that can be determined by the boundary determiner 806. The boundary determiner 806 can determine time periods, such as the time periods 314, 316, 318, 320, 322, 324, 326 shown and described with respect to FIG. 3, based on the determined boundaries.

[0078] In some examples, the boundary determiner 806 determines the boundaries, which can be considered temporal boundaries, based on a determination that the current symbol being writtenis different than a symbol that was previously written. The determination that the current symbol being written is different than a symbol that was previously written is based on the measurements processed by the measurement processor 802.

[0079] In some examples, the boundary determiner 806 determines the boundary based on measurements, such as accelerometer measurements, in a direction (such as the z direction shown and described with respect to FIG. 3) that is orthogonal to a two-dimensional plane in which a majority of movements (such as movements in the x direction and y direction shown and described with respect to FIG. 3) were measured. In some examples, the boundary determiner 806 determines the boundary based on an increase in value (and / or absolute value) of measurements in the direction orthogonal to the two-dimensional plane in which a majority of movements were previously measured. The measurements indicating movement in the orthogonal direction can indicate, and / or be based on, the user 102 lifting the electronic stylus 108 to write a new symbol such as a new letter.

[0080] The computing device 800 can include a writing generator 808. The writing generator 808 can generate writing based on the measurements processed by the measurement processor 802 and the symbols determined by the symbol determiner 804. The writing generator 808 can generate writing based on the series of measurements processed by the measurement processor 802 and the symbols determined by the symbol determiner 804 and based on a neural network that has been trained as discussed above with respect to FIGs. 4 and 6.

[0081] The writing generator 808 can include a stroke augm enter 810. The stroke augmenter 810 can augment and / or modify strokes. The augmentation and / or modification can include changing the generated and / or displayed writing from how the writing would be generated and / or displayed based only on the measurements processed by the measurement processor 802. The augmentations and / or modifications can be based on the current symbol that is determined by the symbol determiner 804. The augmentations and / or modifications can, for example, eliminate stray movements caused by jitter in the electronic stylus 108. In some examples, the augmentations and / or modifications can be based on a trained model that correlates measured movements to determined locations of contact by the electronic stylus 108 with the surface. An example of the training of the model is the correlation of a series of measured movements (such as three-axis accelerometer measurements 606 and / or three-axis gyroscope measurements 608) with determined locations of contacts on a capacitive surface 602, as shown and described withrespect to FIG. 6. Tn some examples, the augmentations and / or modifications can be based on predetermined strokes that form symbols, causing the generated and / or displayed writing to be more similar to the determined symbol than would be the case if the writing were generated and / or displayed based only on the received measurements.

[0082] The writing generator 808 can include a writing modifier 812. The writing modifier 812 can modify writing that has already been presented and / or displayed. The writing that has already been presented and / or displayed can be considered initial writing. The writing modifier 812 can modify the writing based on determining that a more-recent determination of how the writing should be displayed is more accurate than, and / or has a higher confidence level than, the determination of how the writing should be displayed based on which the writing was displayed. Writing presented after the initial writing has been presented and / or modified can be considered additional writing. The writing modifier 812 can determine that the writing should be modified based on the determination that the current determination of how the writing should be displayed is more accurate than, and / or has a greater confidence level than, the previous determination of how the writing should be displayed. In some examples, the more-recent determination can be more accurate and / or have a greater confidence level based on additional measurements providing information based on which the symbol determiner 804 can determine the current symbol.

[0083] In some examples, the writing modifier 812 determines that the presentation of the writing should be modified based on a greater accuracy and / or confidence level of the determined writing and a time that has elapsed since displaying the writing and / or portion of the stroke. The time may have satisfied a time threshold, such as being less than or equal to a time threshold. In some examples, the writing modifier 812 determines that the presentation of the writing should be modified based on the greater accuracy and / or confidence level of the determined writing satisfying a confidence threshold. In some examples, the confidence threshold represents an improvement and / or increase in the accuracy and / or confidence level in the determined writing. In some examples, the confidence threshold is based on a time that has elapsed since displaying the writing and / or portion of the stroke, as discussed above with respect to FIG. 7.

[0084] In some examples, the computing device 800 includes a location determiner 814. The location determiner 814 determines an initial location (such as the initial location 118 shown anddescribed with respect to FIG. 1) of the electronic stylus 108. The location determiner 814 can determine the initial location of the electronic stylus 108 based, for example, on images captured by a camera or other sensing device included in, and / or in communication with, the computing device 800. The computing device 800 can determine an initial location of, and / or starting point for, writing, based on the initial location determined by the location determiner 814. The computing device 800 can display the writing based on the movements inferred from the measurements with the determined location as the starting point for the writing.

[0085] The computing device 800 can include an input trigger 816. The input trigger 816 can determine when the computing device 800 should start generating writing. Examples of writing 116A, 116B, 556 are shown and described with respect to FIGs. 5A and 5B. The writing is based on measurements that indicate and / or are caused by movement of the electronic stylus 108. The trigger based on which the input trigger 816 determines that the computing device 800 should start generating writing can include, for example, a determination that the user 102 has pressed or otherwise activated a button or other input device on the electronic stylus 108. The computing device 800 can respond to the activation of the button on the electronic stylus 108 by generating writing based on measured movements of the electronic stylus 108. In some examples, the input trigger 816 can determine that the generation of writing should begin based on a determination that the electronic stylus 108 has contacted a surface. The input trigger 816 can determine that the electronic stylus 108 has contacted a surface based, for example, on acceleration measurements indicating that the electronic stylus 108 has stopped suddenly.

[0086] The computing device 800 can include at least one processor 818. The at least one processor 818 can execute instructions, such as instructions stored in at least one memory device 820, to cause the computing device 800 to perform any combination of methods, functions, and / or techniques described herein.

[0087] The computing device 800 can include at least one memory device 820. The at least one memory device 820 can include a non-transitory computer-readable storage medium. The at least one memory device 820 can store data and instructions thereon that, when executed by at least one processor, such as the processor 818, are configured to cause the computing device 800 to perform any combination of methods, functions, and / or techniques described herein. Accordingly, in any of the implementations described herein (even if not explicitly noted in connection with a particular implementation), software (e g., processing modules, storedinstructions) and / or hardware (e g., processor, memory devices, etc.) associated with, or included in, the computing device 800 can be configured to perform, alone, or in combination with the computing device 800, any combination of methods, functions, and / or techniques described herein.

[0088] The computing device 800 may include at least one input / output node 822. The at least one input / output node 822 may receive and / or send data, such as from and / or to, a server, and / or may receive input and provide output from and to a user. The input and output functions may be combined into a single node, or may be divided into separate input and output nodes. The input / output node 822 can include, for example, a microphone, one or more cameras, an IMU, one or more displays, a speaker, a microphone, one or more buttons, and / or one or more wired or wireless interfaces for communicating with other computing devices such as the electronic stylus 108.

[0089] FIGs. 9A, 9B, and 9C show an example of the head-mounted device 104. As shown in FIGs. 9A, 9B, and 9C, the example head-mounted device 104 includes a frame 902. The frame 902 includes a front frame portion defined by rim portions 903 A, 903B surrounding respective optical portions in the form of lenses 502A, 502B, with a bridge portion 924 connecting the rim portions 903 A, 903B. Arm portions 905 A, 905B are coupled, for example, pivotably or rotatably coupled, to the front frame by hinge portions 910A, 910B at the respective rim portion 903 A, 903B. In some examples, the lenses 502A, 502B may be corrective / prescription lenses. In some examples, the lenses 502A, 502B may be an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters. Displays 912A, 912B, which can present and / or display writing such as writing 116A, 116B shown in FIG. 5A, may be coupled in a portion of the frame 902. In the example shown in FIG. 9B, the displays 912A, 912B are coupled in the arm portions 905A, 905B and / or rim portions 903 A, 903B of the frame 902. In some examples, the head-mounted device 104 can also include an audio output device 916 (such as, for example, one or more speakers), an illumination device 918, at least one processor 911, an outward-facing image sensor 914 (or camera), and gaze-tracking cameras 919A, 919B that can capture images of eyes of the user 102 to track a gaze of the user 102. In some examples, the head-mounted device 104 may include a see-through near-eye display. For example, the displays 912A, 912B may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e g., 30-45degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world, for example, through the lenses 502A, 502B, next to content (for example, digital images, user interface elements, virtual content, and the like) generated by the displays 912A, 912B. In some implementations, waveguide optics may be used to depict content on the displays 912A, 912B via outcoupled light 920A, 920B. The images projected by the displays 912A, 912B onto the lenses 502A, 502B may be translucent, allowing the user 102 to see the images projected by the displays 912A, 912B as well as physical objects beyond the head-mounted device 104.

[0090] FIG. 10 is a flowchart showing a method 1000 performed by a computing device according to an example. The computing device can include the computing device 800. The method 800 includes receiving measurements (1002). Receiving measurements (1002) can include receiving, from an electronic stylus, a series of three-dimensional accelerometer measurements of movement of the electronic stylus. The method 1000 can include determining a symbol (1004). Determining a symbol (1004) can include determining, based on the series of three-dimensional accelerometer measurements, a symbol being written. The method 1000 can include determining writing (1006). Determining writing (1006) can include generating two- dimensional writing based on the series of three-dimensional accelerometer measurements and the determined symbol.

[0091] In some examples, generating two-dimensional writing includes determining a first stroke of the two-dimensional writing based on a first set of the series of three-dimensional accelerometer measurements and the determined symbol; and determining a second stroke of the two-dimensional writing based on a second set of the series of three-dimensional accelerometer measurements and the determined symbol, the second set being measured after the first set.

[0092] In some examples, the method 1000 further includes presenting, within an augmented reality environment, virtual writing based on the generated two-dimensional writing.

[0093] In some examples, the method 1000 further includes determining an initial location of the electronic stylus based on at least one image captured by a camera. Presenting the virtual writing within the augmented reality environment is based on the initial location and the generated two- dimensional writing.

[0094] Tn some examples, the method 1000 further includes, before determining the symbol being written, causing a portion of a stroke of the two-dimensional writing to be displayed based on the received series of three-dimensional accelerometer measurements; and based on the determining the symbol being written, changing the display of the portion of the stroke. The stroke represents a component of the symbol being written.

[0095] In some examples, the method 1000 further includes determining to change the display of the portion of the stroke based on a time that has elapsed since displaying the portion of the stroke.

[0096] In some examples, the method 1000 further includes determining to change the display of the portion of the stroke based on a confidence level of the change to the display of the portion of the stroke.

[0097] In some examples, the method 1000 further includes determining to change the display of the portion of the stroke based on a time that has elapsed since displaying the portion of the stroke, and a confidence level of the change to the display of the portion of the stroke.

[0098] In some examples, the symbol includes an alphanumeric character.

[0099] In some examples, the symbol includes a geometric shape.

[0100] In some examples, the movement of the electronic stylus includes movement along a surface.

[0101] In some examples, the movement of the electronic stylus includes movement along a flat surface.

[0102] In some examples, the series of three-dimensional accelerometer measurements includes a first set of accelerometer measurements measured during a first time period and the symbol is a first symbol, and the method 1000 further includes receiving a second series of accelerometer measurements from the electronic stylus, the second series of multiple accelerometer measurements being measured during a second time period after the first series of accelerometer measurements and during movement of the electronic stylus; determining, based on the second series of accelerometer measurements, a second symbol being written; and generating additional two-dimensional writing based on the second series of accelerometer measurements.

[0103] In some examples, the method 1000 further includes determining, based on the first series of accelerometer measurements and the second series of accelerometer measurements, that a different symbol is being written during the first time period than during the second timeperiod; and determining a temporal boundary between the first time period and the second time period based on the determining that the different symbol is being written during the first time period than during the second time period.

[0104] In some examples, determining the temporal boundary is based on accelerometer measurements included in the first series of accelerometer measurements and the second series of accelerometer measurements near the temporal boundary being in a direction orthogonal to a two-dimensional plane in which a majority of movements were measured by the accelerometer measurements.

[0105] In some examples, the determining the symbol being written is performed in response to activation of an input sensor included in the electronic stylus.

[0106] FIG. 11 is a flowchart showing a method 1100 performed by a computing device according to an example. The computing device can include the computing device 800. The method 1100 can include receiving a first series of measurements (1102). Receiving a first series of measurements (1102) can include receiving, from an electronic stylus, a first series of accelerometer measurements. The method 1100 can include presenting initial writing (1104). Presenting initial writing (1104) can include presenting initial writing based on the first series of accelerometer measurements. The method 1100 can include receiving a second series of measurements (1106). Receiving a second series of measurements (1106) can include receiving a second series of accelerometer measurements. The method 1100 can include determining a symbol (1108) Determining a symbol (1108) can include determining a symbol based on the first series of accelerometer measurements and the second series of accelerometer measurements. The method 1100 can include modifying writing (1110). Modifying writing (1110) can include modifying the initial writing based on determining the symbol. The method 1100 can include presenting additional writing (1112). Presenting additional writing (1112) can include presenting additional writing based on the symbol and the second series of accelerometer measurements.

[0107] In some examples, modifying the initial writing includes modifying the initial writing based on determining the symbol, and determining that a time period between presenting the initial writing and determining the symbol satisfies a time threshold.

[0108] In some examples, modifying the initial writing includes modifying the initial writing based on determining the symbol, and determining that a confidence level associated with modifying the initial writing satisfies a confidence threshold

[0109] Tn some examples, modifying the initial writing includes modifying the initial writing based on determining the symbol; and determining that a confidence level associated with modifying the initial writing satisfies a confidence threshold, the confidence level threshold being based on a time period between presenting the initial writing and determining the symbol.

[0110] FIG. 12 is a flowchart showing a method 1200 performed by a computing device according to an example. The computing device can include the computing device 800. The method 1200 can include receiving a first series of measurements (1202). Receiving a first series of measurements (1202) can include receiving, from an electronic stylus during a first time period, a first series of accelerometer measurements. The method 1200 can include determining a first symbol (1204). Determining a first symbol (12004) can include determining a first symbol based on the first series of accelerometer measurements;. The method 1200 can include generating first writing (1206). Generating first writing (1206) can include generating first writing based on the first series of accelerometer measurements and the first symbol. The method 1200 can include receiving a second series of measurements (1208). Receiving a second series of measurements can include receiving, from the electronic stylus during a second time period, a second series of accelerometer measurements, the second time period being after the first time period. The method 1200 can include determining a second symbol (1210). Determining a second symbol can include determining a second symbol based on the second series of accelerometer measurements, the second symbol being different than the first symbol. The method 1200 can include generating second writing (1212). Generating second writing can include generating second writing based on the second series of accelerometer measurements and the second symbol.

[0111] In some examples, determining the second symbol includes determining the second symbol based on the second series of accelerometer measurements and the first symbol.

[0112] Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, includingcompiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0113] Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0114] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.

[0115] To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0116] Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0117] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the embodiments of the invention.

Claims

WHAT TS CLAIMED IS:

1. A method performed by a computing device, the method comprising: receiving, from an electronic stylus, a series of three-dimensional accelerometer measurements of movement of the electronic stylus; determining, based on the series of three-dimensional accelerometer measurements, a symbol being written; and generating two-dimensional writing based on the series of three-dimensional accelerometer measurements and the determined symbol.

2. The method of claim 1, wherein the generating two-dimensional writing includes: determining a first stroke of the two-dimensional writing based on a first set of the series of three-dimensional accelerometer measurements and the determined symbol; and determining a second stroke of the two-dimensional writing based on a second set of the series of three-dimensional accelerometer measurements and the determined symbol, the second set being measured after the first set.

3. The method of claim 1 or 2, further comprising presenting, within an augmented reality environment, virtual writing based on the generated two-dimensional writing.

4. The method of claim 3, further comprising: determining an initial location of the electronic stylus based on at least one image captured by a camera, wherein presenting the virtual writing within the augmented reality environment is based on the initial location and the generated two-dimensional writing.

5. The method of any one of the preceding claims, further comprising: before determining the symbol being written, causing a portion of a stroke of the two- dimensional writing to be displayed based on the received series of three-dimensional accelerometer measurements; and based on the determining the symbol being written, changing the display of the portion of the stroke,wherein the stroke represents a component of the symbol being written.

6. The method of claim 5, further comprising determining to change the display of the portion of the stroke based on a time that has elapsed since displaying the portion of the stroke.

7. The method of claim 5 or 6, further comprising determining to change the display of the portion of the stroke based on a confidence level of the change to the display of the portion of the stroke.

8. The method of claim 5, further comprising determining to change the display of the portion of the stroke based on: a time that has elapsed since displaying the portion of the stroke; and a confidence level of the change to the display of the portion of the stroke.

9. The method of any one of the preceding claims, wherein the symbol includes an alphanumeric character.

10. The method of any one of the preceding claims, wherein the symbol includes a geometric shape.

11. The method of any one of the preceding claims, wherein the movement of the electronic stylus includes movement along a surface.

12. The method of any one of the preceding claims, wherein the movement of the electronic stylus includes movement along a flat surface.

13. The method of any one of the preceding claims, wherein: the series of three-dimensional accelerometer measurements includes a first set of accelerometer measurements measured during a first time period and the symbol is a first symbol, and the method further comprises:receiving a second series of accelerometer measurements from the electronic stylus, the second series of multiple accelerometer measurements being measured during a second time period after the first series of accelerometer measurements and during movement of the electronic stylus; determining, based on the second series of accelerometer measurements, a second symbol being written; and generating additional two-dimensional writing based on the second series of accelerometer measurements.

14. The method of claim 13, further comprising: determining, based on the first series of accelerometer measurements and the second series of accelerometer measurements, that a different symbol is being written during the first time period than during the second time period; and determining a temporal boundary between the first time period and the second time period based on the determining that the different symbol is being written during the first time period than during the second time period.

15. The method of claim 14, wherein determining the temporal boundary is based on accelerometer measurements included in the first series of accelerometer measurements and the second series of accelerometer measurements near the temporal boundary being in a direction orthogonal to a two-dimensional plane in which a majority of movements were measured by the accelerometer measurements.

16. The method of claim 1, wherein the determining the symbol being written is performed in response to activation of an input sensor included in the electronic stylus.

17. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device to: receive, from an electronic stylus, a first series of accelerometer measurements; present initial writing based on the first series of accelerometer measurements;receive a second series of accelerometer measurements; determine a symbol based on the first series of accelerometer measurements and the second series of accelerometer measurements; modify the initial writing based on determining the symbol; and present additional writing based on the symbol and the second series of accelerometer measurements.

18. The non-transitory computer-readable storage medium of claim 17, wherein modifying the initial writing includes modifying the initial writing based on: determining the symbol; and determining that a time period between presenting the initial writing and determining the symbol satisfies a time threshold.

19. The non-transitory computer-readable storage medium of claim 17 or 18, wherein modifying the initial writing includes: modifying the initial writing based on determining the symbol; and determining that a confidence level associated with modifying the initial writing satisfies a confidence threshold.

20. The non-transitory computer-readable storage medium of any one of claims 17-19, wherein modifying the initial writing includes: modifying the initial writing based on determining the symbol; and determining that a confidence level associated with modifying the initial writing satisfies a confidence threshold, the confidence level threshold being based on a time period between presenting the initial writing and determining the symbol.

21. A computing device comprising: at least one processor; and a non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by the at least one processor, cause the computing device to:receive, from an electronic stylus during a first time period, a first series of accelerometer measurements; determine a first symbol based on the first series of accelerometer measurements; generate first writing based on the first series of accelerometer measurements and the first symbol; receive, from the electronic stylus during a second time period, a second series of accelerometer measurements, the second time period being after the first time period; determine a second symbol based on the second series of accelerometer measurements, the second symbol being different than the first symbol; and generate second writing based on the second series of accelerometer measurements and the second symbol.

22. The computing device of claim 21, wherein determining the second symbol includes determining the second symbol based on the second series of accelerometer measurements and the first symbol.

Images