Writing device with electromagnetic tracking
By using the combination of electromagnetic attitude tracking system and auxiliary sensors in computer input devices, the limited problem of equipment's attitude recognition in three-dimensional space is solved, efficient use and portability in VR and AR applications are achieved, and user experience is improved .
Patent Information
- Application Number
- CN201980066373.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-09
- Filing Date
- 2019-10-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-10-09
AI Technical Summary
Existing pen/stylus-shaped computer input devices have limited position tracking capabilities in two-dimensional space, limiting their use in 3D space in virtual reality (VR) and augmented reality (AR) applications and often require specialized The surface is operated, affecting its portability and flexibility.
Using an electromagnetic (EM) attitude tracking system, the device's attitude is recognized and the six-degree of freedom is supported by embedding transmitter and receiver modules in the input device, without the need for sight and special surfaces, combined with auxiliary sensors to generate weighted attitudes, enhance the device flexibility and user experience.
Accurate posture recognition and tracking in three-dimensional space, enhances the portability and flexibility of the device, supports the use of VR and AR applications, and maintains the shape factor of the pen or stylus, improving the user experience.
Smart Images

Figure CN113039511B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to input devices for computing platforms, and more particularly to input devices having a stylus or pen form factor. Background Art
[0002] The form factor of a computer input device may have a considerable impact on the user's experience of a computing platform. Therefore, in order to enhance the user experience, computer input devices with pen or stylus form factors have been developed. These devices allow the user to keep the input device in a comfortable position familiar to the user (i.e., a comfortable writing position). In addition, devices with a pen / stylus form factor support relatively precise control by the user, thereby improving the user's experience of various applications such as drawing and painting applications, writing applications, etc. However, conventional input devices with a pen / stylus form factor typically have limited position tracking in two dimensions, thereby limiting the utility of the device for applications such as virtual reality (VR) and augmented reality (AR) applications that use three-dimensional space. In addition, conventional pen / stylus input devices sometimes require the user to write on a special surface such as a specially designed tablet or paper in order to track the position of the input device. These special surfaces limit the portability and flexibility of the input device. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] The present disclosure may be better understood, and its numerous features and advantages become apparent to those skilled in the art by referencing the accompanying drawings.The use of the same reference numbers in different drawings indicates similar or identical items.
[0004] Figure 1 is a block diagram of an electromagnetic (EM) gesture tracking system for a writing device employed for computer input, in accordance with some embodiments.
[0005] Figure 2 is a block diagram of another example of an EM pose tracking system for a writing device employed for computer input in accordance with some embodiments.
[0006] Figure 3 is a diagram illustrating the ability to Figure 1 and Figure 2 Block diagram of the EM pose tracking system used in conjunction with the writing device.
[0007] Figure 4 is a block diagram illustrating an example of an EM pose tracking system with a writing device using a single coil in the writing device in accordance with some embodiments.
[0008] Figure 5 is a block diagram illustrating an example of employing an EM gesture tracking system for a writing device in conjunction with a portable device such as a smart phone in accordance with some embodiments.
[0009] Figure 6 is an illustration of the use of some embodiments that can be Figure 1 and Figure 2 Block diagram of the EM position tracking system in conjunction with a writing device using a force sensor.
[0010] Figure 7 is an illustration of the use of some embodiments that can be Figure 1 and Figure 2 Block diagram of an EM position tracking system in conjunction with a writing device using a rotation sensor.
[0011] Figure 8 is an illustration of the use of some embodiments that can be Figure 1 and Figure 2 Block diagram of an EM position tracking system in conjunction with a writing device using a light sensor.
[0012] Fig. 9 is an illustration of the use of some embodiments that can be Figure 1 and Figure 2 Block diagram of the EM position tracking system in conjunction with a writing device using a pluggable dongle.
[0013] Fig.10 is a diagram illustrating the use of Figure 1 Figure 2. Example of an EM position tracking system used to detect handwriting orientation.
[0014] Fig.11 is a diagram illustrating the use of Figure 1 Flow chart of a method for using an EM position tracking system to identify the orientation of handwriting on a surface for handwriting recognition. DETAILED DESCRIPTION
[0015] Figure 1-11The illustration uses a technology of an electromagnetic (EM) gesture tracking system with a computer input device having a pen or stylus form factor. In some embodiments, the base station device includes one of a transmitter (Tx) module or a receiver (Rx) module for the EM gesture tracking system, while the computer input device includes the other of a TX module and a receiver module. The EM gesture tracking system uses a Tx module and an Rx module to collect EM gesture data indicating the relative gesture between the Tx module and the Rx module. Based on the EM gesture data, the EM gesture tracking system (or a computer device working with the EM gesture tracking system) identifies the gesture (position, orientation, or both position and orientation) of the computer input device. In some embodiments, gestures are identified in a six-degree-of-freedom (6DoF) space, thereby supporting user input for applications such as VR or AR applications that employ such a space. In addition, the EM gesture tracking system supports the recognition of gestures without requiring a line of sight between the base station and the input device and without requiring a special writing surface, thereby enhancing the flexibility of the input device. In addition, components such as EM coils of the EM gesture tracking system can be inserted into the input device without a substantial increase in the size of the input device, thereby allowing the input device to maintain a pen or stylus form factor and thereby enhancing the user experience.
[0016] For convenience, the term "writing device" is used herein to refer to a computer input device having a pen or stylus form factor as further described herein. It will be appreciated that in some embodiments, the writing device can include a writing material, such as an ink reservoir or graphite, that supports writing on a medium (e.g., paper) independent of gesture tracking. In other embodiments, the writing device does not include a writing material, but includes a nub or tip that supports use of the device on a display screen or other surface. In addition, it will be appreciated that the writing device can be used for purposes other than handwriting, including drawing, selecting, and manipulating objects in computer applications including CAD, AR, or VR applications, or any other use in which a stylus can provide input to a computer.
[0017] Figure 1The tracking system 100 that employs EM gestures to recognize the gestures of a writing device illustrated as stylus 102 is illustrated. In the depicted example, the tracking system 100 includes a base device 104 and stylus 102. The base device 104 may be part of or connected to a computer device, may be a base station independent of the computer device, or may be a mobile device in some embodiments. As further described herein, the tracking system 100 is generally configured to recognize the gesture of the stylus 102 based on a weighted combination of an EM gesture (a gesture generated based on an EM field 125 as further described herein) and an auxiliary gesture (a gesture generated based on data provided by an auxiliary sensor 106). The tracking system 100 can therefore be incorporated into various devices and systems that employ gesture recognition features. For example, in some embodiments the tracking system 100 is incorporated into a virtual reality (VR) system to recognize the gesture of the stylus 102 to allow a user to manipulate objects in a VR environment using the stylus 102. Thus, in some configurations, the base device 104 is a head mounted display (HMD) that displays a VR environment to a user. In other configurations, the base device 104 is a device separate from the HMD (such as an accessory or base station). It should be appreciated that other configurations of the tracking system 100 can be employed. For example, in some embodiments the tracking system 100 does not employ the secondary sensor 106 and generates the pose of the stylus 102 based solely on the EM pose data.
[0018] To generate an EM pose, tracking system 100 generates an EM field 125, measures the amplitude and / or phase of the generated EM field 125 (generally referred to herein as "EM data"), and calculates a relative pose based on the corresponding EM data. In the illustrated embodiment, EM field 125 is generated by base device 104 and the EM data is read at stylus 102. It should be appreciated that other configurations are possible, including EM field 125 generated by stylus 102 and EM data read at base device 104, as described below in Figure 21. To support the generation of the EM field, the base device 104 includes a Tx module 121, whereby the Tx module 121 includes a transmitter coil 116 and an amplifier 114. In some embodiments, the transmitter coil 116 is a three-axis coil that is generally configured to generate an EM field 125 at an intensity referred to herein as a transmit power, wherein the transmit power is based on the electrical power provided to the transmitter coil 116 by the amplifier 114. The amplifier 114 is a programmable amplifier that is generally configured to generate electrical power at an amplitude based on received control signaling. Therefore, the transmit power for the EM field 125 is a programmable value controlled at the base device 104. The Tx module also includes a digital-to-analog DAC converter 112 that converts a value provided by the CPU 115 of the base device 104 to supply input to the amplifier 114. In other embodiments, the amplifier 114 is a non-programmable amplifier and is able to change the amplitude of the EM field simply by reducing the amplitude of the source signal, which may be done digitally or via a DAC. CPU 115 can perform other operations on behalf of base device 104 and tracking system 100, including generating gestures of stylus 102 as further described herein. In other embodiments, one or more of these operations can be performed by CPU 110 or by a third CPU.
[0019] To support the reading of EM data, stylus 102 includes an Rx module 120 having a receiver coil 107 and an analog-to-digital converter (ADC) 108. In some embodiments, receiver coil 107 is a three-axis coil configured to generate an analog electrical signal having an amplitude and / or phase indicative of a detected EM field 125. Figure 4 In some embodiments further described, the receiver coil 107 may have fewer than three coils. The ADC 108 is generally configured to receive the generated analog signal and convert the analog signal into a digital value indicative of the analog signal and, therefore, the detected EM field 125. Figure 1 In the example of , the digital value generated by ADC 108 is EM data that can be used for gesture recognition as described further below.
[0020] To support the generation of gestures, stylus 102 includes an auxiliary sensor 106 configured to generate non-EM gesture information of stylus 102. In some embodiments, the auxiliary sensor is an inertial measurement unit (IMU). In other embodiments, the auxiliary sensor is an auxiliary tracking system (in Figure 1 The auxiliary tracking system is a part of the auxiliary tracking system (not shown in the figure), which generates auxiliary posture information based on auxiliary tracking signals such as light-based signals (e.g., laser, LED light, etc.), sound wave signals (e.g., ultrasonic signals) and radio frequency (RF) signals.
[0021] To support gesture recognition (i.e., recognition of gestures of the base device 104 or the mobile device), the stylus 102 and the base device 104 include processors 110 and 115, respectively. Processors 110 and 115 are general or special-purpose processors that are generally configured to execute instructions (e.g., computer programs) in order to perform specific tasks. In some embodiments, at least one of the processors 110 and 115 executes instructions to recognize a gesture associated with the base device 104 or the mobile device based on a weighted combination of EM data provided by the ADC 108 and secondary gesture data generated by the secondary sensor 106. In other embodiments, a different processor on an external system (e.g., such as a PC computer) is used to calculate the gesture. In one example, in some embodiments, the processor 110 recognizes the gesture based on the following formula:
[0022] pose final =(W em *pose em +W secondary *pose secondary ) / (W em +W secondary )
[0023] Where W em is the weight applied to the pose calculated from the EM signal, pose em is the pose calculated from the EM data, W secondary is the weight applied to the pose calculated from the auxiliary tracking signal, and pose secondary is the attitude calculated based on the auxiliary tracking signal.
[0024] In addition, Figure 1 In the example of , processors 110 and 115 are connected via a communication link 128 to support communication of EM data, auxiliary gesture data, recognized gestures, or other information. For example, in some embodiments, processor 110 recognizes gestures based on EM data and auxiliary gesture data and passes the recognized gestures to processor 115. In other embodiments, processor 110 passes EM data and auxiliary gesture data to processor 115, which recognizes gestures based on weighted combinations. Communication link 128 can be a wired communication link, a wireless communication link (e.g., Bluetooth), etc., or a combination thereof. In other embodiments, EM data and auxiliary gesture data can be sent to a third processor (not shown), where the gesture is calculated based on a weighted combination of the EM data and the auxiliary gesture data.
[0025] In some embodiments, one or more of processors 110 and 115 (or Figure 1A third processor (not shown in the figure) executes an additional instruction set to use the posture determined based on the EM data and the auxiliary posture data. For example, in some embodiments, the processor 115 can execute the instruction set to display the virtual reality environment to the user and use the recognized posture to determine the position of the stylus 102 in the virtual reality environment, thereby allowing the user to use the stylus 102 to interact with the virtual reality environment. In some embodiments, the stylus 102 is used to capture handwriting or hand drawings. The user can write or draw on paper (for example, with ink or pencil tip) and the writing or drawing is captured by the computing device based on the posture of the stylus 102, thereby allowing the writing or drawing to be displayed at the computing device. In other embodiments, the stylus 102 is used as a stylus on a computer screen in a typical stylus use case. In these embodiments, the stylus may be used for annotation, writing, or as a regular mouse input. In still other embodiments, the stylus 102 can be used as a 3D mouse for manipulating 3D virtual objects. For example, the stylus 102 can replace the "space mouse" for manipulating the rotation or displacement of a 3D CAD drawing.
[0026] As noted above, in some embodiments, the Tx module and the Rx module of the tracking system 100 are relatively Figure 1 In the opposite position. Figure 2 An example is shown in the figure. Figure 2 A tracking system 200 is shown in accordance with some embodiments. The tracking system 200 includes components similar to those described above with respect to the tracking system 100. However, for the tracking system 100, the TX module 121 is located at the stylus 102, while the Rx module 120 is located at the base device 104. The components are similar to Figure 1 The corresponding components of are executed to support the generation of gestures of the stylus 102.
[0027] It should be appreciated that stylus 102 may be any type of writing device. Figure 3An example embodiment of a stylus 102 is illustrated that can be employed to write on paper or other media while concurrently generating gestures of the stylus 102. In the depicted example, the stylus 102 includes both an EM coil 107 to support the generation of EM gestures and also includes a writing material 328. The writing material 328 can be any kind of writing material that is dispensed from the tip 323 of the stylus 102 onto a writing surface (e.g., paper or other media) as the user moves the stylus 102. Thus, in some embodiments the writing material 328 is ink contained in a reservoir, wherein both the reservoir and the EM coil are placed within a housing that forms the exterior of the stylus 102. In other embodiments, the stylus 102 is a pencil and the writing material 328 is graphite contained within the housing of the stylus 102 and dispensed at the tip 323 of the stylus 102. In other embodiments, the stylus 102 includes a non-writing tip that is configured to be placed on a computer or tablet screen or other surface.
[0028] In some cases, metal or other materials located in close proximity to Rx module 120 or to Tx module 121 can cause distortions in EM field 125, and these distortions can produce inaccuracies in the EM gesture data. For example, if stylus 102 is being used to write on a tablet containing metal, such that tip 323 is close to the metal, the resulting distortions in EM field 125 can cause errors in the EM gesture data, resulting in gesture tracking errors for stylus 102, and, in turn, a poor user experience. However, distortions in the EM field can be mitigated by placing EM coil 107 at a distance from the surface that is causing the distortion. For example, in a device that includes Figure 3 In some embodiments of the example embodiments of the present invention, the EM coil 107 is placed relatively close to the end 324 of the stylus 102, which is opposite the tip 323. The EM coil 107 is thus relatively far away from the distorted surface when the tip 323 is in contact with the surface, thereby reducing distortion in the EM field 125 and thereby improving the accuracy of the EM gesture data.
[0029] In some cases, it may be desirable to reduce the cost or size of the circuitry in stylus 102. Figure 4 An example is illustrated in FIG. In the depicted embodiment, the stylus 102 and the base device 104 are similar to Figure 2Embodiments are configured. However, stylus 102 employs EM coil 430 wrapped around shaft 433 along the length of stylus 102, which allows EM coil 430 to be placed within the housing of stylus 102 with relatively little impact on the form factor of stylus 102. In some embodiments, EM coil 430 is the only coil employed by the Rx module. In other embodiments, the Rx module employs an EM coil in addition to EM coil 430. Furthermore, by winding coil 432 along the length of stylus 102, coil 432 can be implemented with a relatively high number of turns. As understood in the art, the strength of EM field 125 is proportional to the number of turns in the coil, so that a long coil wrapped around stylus 102 allows for a higher signal-to-noise ratio in the EM data while having a relatively small impact on the size of stylus 102.
[0030] In some embodiments, the tracking system can reduce the overall cost of the system by utilizing existing EM components of a computer, smartphone, or other device. Figure 5 An example is illustrated in FIG. In the depicted example, stylus 102 is similar to Figure 2 541 and a second sensor 106. However, in the illustrated example, the base device is a smartphone 540 that includes a CPU 541 and a magnetometer 542. In some embodiments, the magnetometer 542 is a 3-axis magnetometer embedded in the smartphone 540 and used for different smartphone operations, such as providing a compass-like capability. The CPU 541 is configured to control the magnetometer 542 to follow the same Figure 1-2 107, and in particular collects EM gesture data from the readings of the EM field 125. That is, by using the magnetometer 542 as a base coil for reading the EM signals emitted by the stylus 102, the CPU 541 is able to serve as a "base" and collect EM gesture data from the magnetometer 542, which can then be converted by the CPU 541 into an EM gesture.
[0031] In some embodiments, magnetometer 542 has a relatively low sampling rate (e.g., in the range of 50-200 Hz), which does not readily allow for frequency multiplexing of the EM field, and thus stylus 102 can employ a pulsed EM configuration better suited for these magnetometers. For smartphones with magnetometers or other magnetic sensors with higher sampling rates, the EM field can be multiplexed over different frequencies for simultaneous transmission.
[0032] It should be appreciated that the stylus 102 can also employ different configurations of the stylus tip in addition to or instead of supporting the deposition of writing material on a surface. The different tips can support improved gesture detection, additional stylus functionality, or a combination thereof. In accordance with some embodiments, Figure 6-8Examples of different stylus tip configurations are illustrated in FIG. Figure 6 The stylus 602 is shown to include a CPU 610 and an EM module 620. The EM module 620 may be similar to Figure 1 Rx EM module 120 Rx EM module, or similar Figure 2 Therefore, it is possible to Figure 1-5 Stylus 602 is used in place of stylus 102 in any of the described embodiments.
[0033] The stylus 602 employs a force-sensitive tip configuration including a tip 650, a spring 651, and a force sensor 652, which may be a mechanical force sensor or an electrical force sensor. In some embodiments, the spring 651 is omitted. When the tip 650 contacts a surface (e.g., when a user places the tip 650 on a screen or other writing surface), the force of the surface acting on the tip causes the tip to translate downward, thereby applying a force sensed by the force sensor 652, which indicates the sensed force to the CPU 610. In some embodiments, the force sensor 652 indicates the sensed force to the CPU 610 via a binary signal to indicate whether the force at the tip 650 has exceeded a specified or programmable threshold. In other embodiments, the force sensor 652 indicates the level of the sensed force to the CPU 610 via an electrical (digital or analog) signal, thereby indicating a variable pressure level at the tip 650. Based on the information provided by the force sensor 652, the CPU 610 is able to take one or more specified actions, such as executing an application, changing application parameters (such as the thickness, shading, color, or other aspects of a displayed or stored line), and the like.
[0034] In such Figure 7 In some embodiments of the examples, the stylus can employ a tip for assisting in gesture recognition of the stylus. Figure 7 The stylus 702 is shown including a CPU 710 and an EM module 720. Figure 6 Similarly, the EM module 720 may be similar to Figure 1 Rx EM module 120 Rx EM module, or similar Figure 2 The Tx EM module 121 of the Tx EM module. In addition, it is possible to Figure 1-5In any of the described embodiments, stylus 702 is used in place of stylus 102. Stylus 702 has a tip configuration that acts as a 2-D sensor when dragged against any flat surface, thereby simulating a pen. The configuration includes a ball tip 753 and a pair of roller and rotation sensors 756 and 757. As the user moves the stylus over the surface, the ball tip 753 rotates. The roller and rotation sensor pair 756 and 757 sense the rotation of the ball tip and provide 2-D gesture information to the CPU 710 based on the rotation. For example, in some embodiments the roller and rotation sensor pair 756 indicates the position of the tip 753 along the "y" axis, and the roller and rotation sensor pair 757 indicates the position of the tip 753 along the corresponding "x" axis. The 2-D gesture information can be used to check or augment the gesture identified based on the EM gesture data as described above.
[0035] In such Figure 8 In some embodiments of the examples, instead of tracking the 2-D position of the tip via a mechanical configuration, the stylus can employ optical sensing. Figure 8 The stylus 802 is shown including a CPU 810 and an EM module 820. Figure 6 Similarly, the EM module 820 may be similar to Figure 1 Rx EM module 120 Rx EM module, or similar Figure 2 The Tx EM module 121 of the Tx EM module. In addition, it is possible to Figure 1-5 In any of the described embodiments, stylus 802 is employed in place of stylus 102. Stylus 802 includes a light source 859 and a light sensor 858. Light source 859 is configured to emit a light beam 861 via the tip of stylus 802. Light beam 861 is reflected from a surface 860 external to stylus 802, such as a writing surface, a computer screen, etc. Light sensor 858 uses the reflected light beam to capture images of surface 860 and provides these images to CPU 810. CPU 810 compares each captured image with previously captured images to determine the direction of movement of the stylus across surface 860. CPU 810 can use the identified direction of movement to augment or check a gesture identified based on EM gesture data.
[0036] It should be appreciated that any of the above-described stylus embodiments including stylus 102, 602, 702, and 802 can include additional components for providing additional input or other information to a computer device. For example, the stylus can include one or more buttons to indicate additional input data to a computer device. One or more buttons can be placed in a convenient position for the user, such as, if the stylus is held like a pen or pencil, one of the buttons is placed at or near the user's index finger, while another button is placed at or near the user's thumb. In some embodiments, the button can be placed at the end of the stylus opposite to the tip in a position similar to the button for a retractable ballpoint pen. Additionally, in some embodiments, the stylus can incorporate a slider, 1-D touchpad, roller, or similar component (or combination thereof) that allows one-dimensional linear input. In some embodiments, the 1-D linear input component is placed in front of the stylus near the position where the index finger would normally be placed when held like a pen. In other embodiments, when the stylus is held like a stick, the component is placed closer to the middle for easy access. In other embodiments, the stylus can include a 2-D trackpad, joystick, or similar component that allows 2-D analog input into the stylus. In some embodiments, when the stylus is held like a wand, the 2-D analog input component is placed closer to the middle for easy access.
[0037] In some embodiments, the flexibility and portability of the stylus can be enhanced by allowing the base device 104 to be removably integrated with the stylus. Fig. 9 An example is shown in FIG. Fig. 9 A stylus 902 is shown, which in various embodiments corresponds to any one or combination of the styluses 102 , 602 , 702 , and 802 described above. Fig. 9 Also shown is a dongle 965 including an EM module 966. The EM module 966 may be similar to Figure 1 Tx EM module 121 Rx EM module, or similar Figure 2 The stylus 902 includes an EM module 967 that is complementary to the EM module 966. Thus, if the EM module 966 is a Tx module, the EM module 967 is an Rx module and vice versa. The dongle 965 can thus be mated with the connector 968, making it easy and convenient for the user to carry the dongle 965 with the stylus 902. In some embodiments, the stylus 902 provides power to the dongle 965 from a battery (not shown) or other power source via the connector 968 to charge the battery (not shown) that provides power to the EM module 966.
[0038] The dongle 965 can be inserted into a USB port of a computer device such as a smartphone 940 at USB port 941. The EM module 966 is active when inserted into the USB port 941, thereby transmitting or receiving the EM field 125, thereby allowing the smartphone 940 to operate as described above with respect to Figure 1 The dongle 965 may be configured to recognize gestures based on EM gesture data as described above. In some embodiments, the dongle 965 itself includes a processor to recognize gesture data, thereby reducing the processing load at the smart phone 940. It should be appreciated that other configurations and form factors of the dongle 965 are possible. For example, in some embodiments the dongle 965 has a "cap" form factor that can fit over the tip of the stylus 902.
[0039] In some embodiments, the EM gesture data can be used to assist in handwriting or other text or drawing recognition (collectively referred to herein as "handwriting capture"). For example, in one embodiment, handwriting captured via the EM gesture data is converted from handwriting to typed text using handwriting recognition software or optical character recognition (OCR) software (e.g., by one or both of processors 110 and 115, or by another processor). In another embodiment, the handwriting capture is converted to geometric shapes using shape recognition software. Such processing can "clean up" hand drawings or generate readable variants of charts and graphs. In another embodiment, the handwriting capture is converted to a mathematical equation. Such processing can "clean up" the mathematical equation into a representable digital representation.
[0040] In contrast to conventional handwriting capture that requires the use of specialized surfaces, the use of EM pose data allows handwriting capture on a variety of surfaces. In particular, the use of EM pose data supports handwriting capture when the relative angles or orientations of the surfaces are unknown. For example, a user may write on a wall (where the surface is vertical) and also on a table (where the surface is horizontal), and the angle of the writing plane is not known a priori.
[0041] To support handwriting capture on unknown surfaces, in some embodiments handwriting capture from stylus 102 is "fitted" to a plane to produce an accurate representation of the handwriting. For example, if the writing surface is at a 45 degree angle but the trajectory is projected onto a flat (zero degree angle) plane, the resulting handwriting will appear distorted. To preserve an accurate representation of the handwriting, the plane of the writing surface is determined by a processor, such as processor 110 or processor 115. In one embodiment, the processor employs a plane fitting algorithm to "fit" the writing trajectory to determine the plane angle. In other embodiments, the writing plane is determined "on the fly" based on what the user is writing. In other embodiments, plane fitting is accomplished using post-processing at the processor.
[0042] In some embodiments, the processor determines the angle of the writing based on the direction of the writing. Fig.10 An example is shown in the figure. Fig.10 The diagram shows handwriting 1070 that has been written on a surface with stylus 102. The processor 115 (or other processor) employs a heuristic to determine an angle 1073 relative to a vector 1072 that represents "horizontal" writing (e.g., from left to right or from right to left). The processor 115 uses the angle 1073 to correct the orientation of the handwriting prior to handwriting recognition. For example, in some embodiments the processor 115 fits a line 1071 through the handwriting and measures the angle 1073 between the fitted line and the desired angle. In other embodiments, the orientation of the handwriting can be determined by a machine learning model such as a neural network.
[0043] In some embodiments, the processor performs orientation fitting. For example, handwriting recognition software that can recognize handwriting may cause an error if the handwriting is rotated 180 degrees so that the handwriting is upside down. Therefore, before attempting to recognize the handwriting, the processor 115 determines the "correct side up" orientation of the writing. In some embodiments, the processor attempts to rotate the handwriting by a certain amount and then uses a heuristic to score the expected degree of the resulting rotation. The processor repeats this process until the score exceeds a threshold. In accordance with some embodiments, Fig.11 An example is illustrated at 1102. At frame 1102, processor 115 gives the initial orientation of the handwriting and the initial score based on the initial orientation. For example, in some embodiments, the score can vary in the range of 0.0 to 1.0 and represents the confidence level of the handwriting in the horizontal orientation. At frame 1104, processor 115 determines whether the score exceeds a threshold value. In the above example, the threshold value can be a value of 0.9, indicating that the orientation of the handwriting enables the threshold confidence level of the handwriting to be interpreted by specifying a handwriting recognition algorithm. If the score does not exceed the threshold value, the method moves to frame 1106 and processor 115 rotates the orientation of the handwriting by a specified amount. The method returns to frame 1102 and processor 115 generates a new score based on the adjusted orientation. Processor 115 continues to incrementally adjust the orientation until at frame 1104, the score exceeds the threshold value. In response to the score exceeding the threshold value, the method flow moves from frame 1104 to frame 1108, and processor 115 executes the handwriting recognition algorithm based on the current orientation of the handwriting (the orientation that causes the score to exceed the threshold value).
[0044] In other embodiments, the processor 115 rotates the captured handwriting by multiple angles and processes each rotation in parallel to speed up processing. In some embodiments, scoring is performed by attempting handwriting recognition for some handwriting and using the confidence level from the handwriting recognition algorithm as an input into the score. In still other embodiments, a machine learning model is used to estimate the rotation angles necessary for the handwriting recognition software to properly process the handwriting strokes.
[0045] Embodiments of the present disclosure include fusing multiple data sources to improve the tracking quality of a pen device. In one embodiment, the pose from the EM position tracking subsystem is combined with the pose from the IMU to reduce noise and improve angular accuracy. In some embodiments, a Kalman filter is used to combine the data derived from the IMU with the data derived from the EM position tracking system to produce a "fused" pose. Fusion can also be used to correct temporary distortion caused by nearby metal, for example, by performing "dead reckoning" with the IMU for a short period of time when metal distortion is detected.
[0046] In some embodiments, certain aspects of the above-mentioned techniques may be implemented by one or more processors of a processing system that executes software. The software includes one or more executable instruction sets stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software can include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-mentioned techniques. The non-transitory computer-readable storage medium can include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or one or more other non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium can be in source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by one or more processors.
[0047] Computer-readable storage media may include any storage media or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., compact disk (CD), digital versatile disk (DVD), Blu-ray disk), magnetic media (e.g., floppy disk, tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., network accessible storage (NAS)).
[0048] Note that not all activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more additional activities may be performed, or elements other than the described elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which the activities are performed. In addition, the concept has been described with reference to specific embodiments. However, it will be appreciated by those of ordinary skill in the art that various modifications and variations can be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the specification and the drawings will be considered in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.
[0049] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any features that can make any benefit, advantage, or solution occur or become more significant should not be interpreted as key, essential, or necessary features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter can be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings herein. Except as described in the following claims, it is not intended to be limited to the details of construction or design shown herein. It is therefore apparent that the specific embodiments disclosed above can be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the following claims.
Claims
1. A writing device comprising: an electromagnetic (EM) tracking module configured to generate data indicating a first EM pose of the EM transmitter relative to the EM receiver; a sensor for generating a first sensor pose based on the sensor data; as well as A processor is configured to determine a pose of the writing device based on a weighted combination comprising a first weight applied to the first EM pose and a second weight applied to the first sensor pose, the second weight being different from the first weight.
2. The writing instrument according to claim 1, wherein The EM tracking module includes an EM receiver.
3. The writing instrument according to claim 1, wherein The EM tracking module includes an EM transmitter.
4. The writing instrument of claim 1 , further comprising: A writing implement for depositing material onto a writing surface.
5. The writing instrument according to claim 4, wherein The material includes one of the group consisting of graphite and ink.
6. The writing instrument of claim 1 , wherein: The gestures are employed to track movement of the writing device at the computer device.
7. The writing instrument of claim 1 , wherein: The gesture is employed to capture handwriting at the computer device.
8. The writing instrument of claim 1, wherein The writing device further includes a writing implement including a first end and a second end opposite the first end, and wherein the writing implement is located at the first end and the EM tracking module is located at least 1 centimeter away from the first end.
9. The writing instrument of claim 1, wherein The sensor is selected from the group consisting of an inertial measurement unit, an ultrasonic sensor, an optical sensor, an acoustic wave sensor, and a radio frequency (RF) sensor.
10. The writing instrument of claim 1, wherein The weighted combination includes a sum of the product of the first weight applied to the first EM pose and the first EM pose and the product of the second weight applied to the first sensor pose and the first sensor pose, divided by a sum of the first weight and the second weight.
11. The writing instrument of claim 1 , wherein: The processor processes the data for identifying a gesture of the writing device based on the first EM gesture to manipulate one or more objects in a virtual reality (VR) environment.
12. The writing instrument of claim 1, wherein The electromagnetic EM tracking module measures the amplitude and / or phase of the generated EM field.
13. The writing instrument of claim 1, wherein The EM tracking module includes fewer than three EM coil axes.
14. The writing instrument of claim 13, wherein The writing device includes a stylus, and wherein the EM coil bobbin is wrapped around a long axis of the stylus.
15. The writing instrument of claim 1, wherein The EM tracking module includes a magnetometer.
16. The writing instrument of claim 1 , further comprising: A force sensor is provided for indicating that a force applied to the tip of the writing instrument has exceeded a threshold value.
17. The writing instrument of claim 1 , further comprising: a sensor for indicating a two-dimensional position of a tip of the writing device relative to a writing surface; as well as The processor identifies the posture of the tip based on the two-dimensional position of the tip.
18. The writing instrument of claim 17, wherein The sensor is selected from the group consisting of a roller tracker and an optical tracker.
19. The writing instrument of any one of claims 1 to 18, further comprising: A base device includes one of the EM transmitter and the EM receiver and is removably integrated with the writing device.
20. The writing device of claim 19, wherein the base device further comprises: A dongle includes the other of the EM transmitter and the EM receiver.
21. The writing instrument of claim 20, wherein The dongle is removably coupled to an end of the writing instrument.
22. A method for a handwriting device, comprising: receiving first EM attitude data based on electromagnetic EM field values, the EM field values indicating a first attitude of the EM transmitter relative to the EM receiver; receiving first sensor data indicating a first sensor posture; as well as A pose of the writing device is determined based on a weighted combination including a first weight applied to the first EM pose data and a second weight applied to the first sensor data, the second weight being different from the first weight.
23. The method according to claim 22, wherein The writing instrument includes the EM receiver.
24. The method according to claim 22, wherein The writing device includes the EM emitter.
25. The method of claim 22, further comprising: Movement of the writing device at the computer device is tracked.
26. The method according to claim 22, in, The weighted combination includes the sum of the product of the first weight applied to the first EM pose data and the first EM pose data and the product of the second weight applied to the first sensor data and the first sensor data, divided by the sum of the first weight and the second weight.
27. The method of claim 26, wherein the first sensor is selected from the group consisting of an inertial measurement unit and an ultrasonic sensor.
28. The method of claim 22, further comprising: receiving force data received from a force sensor of the writing device; as well as A determination is made based on the force data as to whether the writing device is contacting a surface.
29. The method according to claim 22, wherein The EM field values are at least partially indicative of the generated EM field, and an amplitude and / or phase of the generated EM field is measured such that the first EM posture data is based on EM amplitude and / or phase field values indicative of the first posture of the EM transmitter relative to the EM receiver.
30. The method of claim 22, further comprising: Handwriting is captured based on a gesture of the writing device.
31. The method of claim 30, further comprising: Fit the captured handwriting to a plane.
32. The method of claim 30, further comprising: The orientation of the handwriting is determined by: iteratively rotating the captured handwriting; For each iteration, an orientation score is determined; as well as An orientation of the captured handwriting is selected in response to the orientation score exceeding a threshold.
33. The method of claim 30, further comprising: determining the orientation of the captured handwriting; as well as An orientation of the captured handwriting is determined based on the direction.
34. The method of claim 30, further comprising: An orientation of the captured handwriting is determined based on a machine learning model.
35. The method according to any one of claims 30 to 34, further comprising: Convert the captured handwriting into digital characters.
