A writing pen touch recognition method based on gesture recognition and inertial sensor

Abstract

The application provides a writing pen touch recognition method based on gesture recognition and inertial sensors, relates to the touch recognition technical field, and is applied to a gesture-pen fusion positioning system. Inertial sensing device data is acquired and processed into the positioning coordinates of a writing pen nib in a three-dimensional space, then into the height information of the pen nib to a writing plane, and finally into the touch recognition result through linkage with the vibration information of the inertial sensing device. The application utilizes the gesture tracking capability commonly used in VR / MR / AR devices on the market, avoids customizing the writing pen, can be embedded in the existing devices at a low cost, is not limited to writing surfaces and writing pen types, has high universality, recognizes touch through the pen nib height auxiliary vibration intensity, and makes the recognition result more accurate and reliable.

Description

Technical Field

[0001] This invention relates to the field of touch recognition technology, and in particular to a pen touch recognition method based on gesture recognition and inertial sensors. Background Technology

[0002] Currently, in virtual reality, mixed reality, and augmented reality technologies, recognizing the writing state of a writing pen is a crucial step in achieving human-computer interaction. Whether a writing pen is in a writing state is primarily determined by whether it is touching the writing surface. Current mainstream touch recognition technologies include both contact-based and non-contact-based touch recognition.

[0003] Contact-based touch recognition methods utilize sensors on the pen tip or writing surface to detect changes in capacitance, resistance, and pressure generated during contact, thereby determining the pen's writing status. This type of technology is the most direct and effective for pen touch recognition and is widely accepted in the market. However, it results in the pen or writing surface being coupled directly to the sensor, leading to drawbacks such as the need for additional device customization and limited writing area, thus exhibiting significant limitations in virtual reality / mixed reality / augmented reality application scenarios.

[0004] Non-contact touch recognition methods primarily rely on optical approaches. These methods calculate the distance from the pen tip to the writing surface using optical sensors on the pen or writing surface, indirectly inferring the pen's touch state. This technology eliminates the need to bind sensors to the pen or writing surface, making recognition possible on any surface or with any pen. However, it still suffers from limitations such as lack of portability and ease of use, and relatively large device size. Summary of the Invention

[0005] The purpose of this invention is to provide a pen touch recognition method based on gesture recognition and inertial sensors, so as to achieve accurate touch recognition of various pens on any surface.

[0006] To achieve the above-mentioned objectives, this invention provides a pen touch recognition method based on gesture recognition and inertial sensors. The method is applied to a gesture-pen fusion positioning system and includes the following steps:

[0007] S101. Collect the acceleration and attitude information of the writing pen through the inertial sensor and upload the acceleration and attitude information to the host.

[0008] S102. The host computer recognizes the writer's gestures, obtains gesture recognition data, and collects its own positioning data.

[0009] S103. The host calculates the three-dimensional coordinates of the pen tip by fusing positioning based on acceleration and attitude information, gesture recognition data, and positioning data.

[0010] S104. Calculate the vibration intensity of the acceleration obtained by the inertial sensing device;

[0011] S105. Determine whether the writing plane initialization has been performed. If not, proceed to step S106. If yes, proceed to step S107.

[0012] S106. Determine a three-dimensional center point on the writing plane and the normal vector of the writing plane to complete the initialization of the writing plane;

[0013] S107. Calculate the height of the pen tip from the writing plane based on the three-dimensional center point and normal vector, and perform joint touch recognition based on vibration intensity and pen tip height until the system stops.

[0014] Furthermore, the gesture-pen fusion positioning system includes an inertial sensor, a writing pen, a fixing strap, and a host. The inertial sensor is used to collect the acceleration and attitude information of the writing pen and transmit the acceleration and attitude information to the host via wireless communication. The inertial sensor is fixed to the tail of the writing pen by the fixing strap, and the direction of the inertial sensor is aligned with the direction of the pen barrel. The host is a computing device with computing power, positioning ability, wireless communication and gesture recognition capabilities.

[0015] Furthermore, the inertial sensing device includes a microprocessor, a MEMS inertial sensor, a wireless communication module, a battery power supply, and physical buttons. The physical buttons are used to send a trigger signal to the microprocessor when pressed. After receiving the trigger signal, the microprocessor extracts the acceleration from the MEMS inertial sensor and the attitude information obtained based on the attitude calculation algorithm, and uploads the acceleration and attitude information to the host via the wireless communication module. The battery power supply is used to provide the energy required for the operation of the inertial sensing device.

[0016] Furthermore, step S103 specifically includes the following operations:

[0017] S201. Based on the vectors pointing from the inertial sensing device to the gripping point at different time nodes, and the sampling interval of the inertial sensing device, a motion model is established, wherein the gripping point is the position where the writer holds the pen body.

[0018] S202. Using the motion model as a framework, Kalman filter fusion positioning is performed to obtain the gripping point displacement;

[0019] S203. Perform vector calculation based on the displacement of the grip point to obtain the three-dimensional coordinates of the pen tip.

[0020] Furthermore, the vibration intensity of the acceleration obtained by the inertial sensing device is calculated, specifically as follows: Let the acceleration collected by the inertial sensor device be... The vibration intensity is the vector magnitude processed by the high-pass filter, expressed as:

[0021]

[0022] In the above formula, For acceleration is Vibration intensity at that time Indicates the cutoff frequency as A high-pass filter.

[0023] Furthermore, step S106 specifically includes the following operations:

[0024] S301. Determine several three-dimensional points on the writing plane. , forming a point set ;

[0025] S302, Calculate the three-dimensional center point of the point set. ;

[0026] S303, Based on point set With the three-dimensional center point Calculate the centered point set and covariance matrix;

[0027] S304. Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and take the eigenvector corresponding to the smallest eigenvalue.

[0028] S305. Let the coordinates of the writer's head be... Calculate the eigenvector and the vector The dot product;

[0029] S306. Calculate the normal vector of the eigenvector based on the value of the dot product, and output the three-dimensional center point and normal vector.

[0030] Furthermore, step S107 specifically includes the following operations:

[0031] S401. Initialize buffers and variables , The buffer is used to record the three-dimensional coordinates of the pen tip in the state to be determined. Used to record the height of the pen tip at the moment of contact with the writing surface. Used to record flag bits;

[0032] S402. Based on the vibration intensity, perform touch recognition to determine whether the writing pen touches the writing surface. If the recognition result is touch, proceed to step S403; otherwise, proceed to step S404.

[0033] S403. Mark the current three-dimensional coordinates of the pen tip as a touch, flag bit. Set to true. Update to current height Mark all three-dimensional coordinates of the pen tip in the buffer as touched and clear the buffer, then jump to step S406;

[0034] S404, Determine the flag bit If true, proceed to step S405; otherwise, mark the current pen tip's three-dimensional coordinates as non-touch and proceed to step S406.

[0035] S405. Calculate the height difference between the current height and the height at the last touch. If the height difference is less than a preset height difference threshold, put the pen tip's three-dimensional coordinates into the buffer; otherwise, set the flag. Set to false, mark all pen tip 3D coordinates in the buffer as non-touched, and clear the buffer;

[0036] S406, Touch recognition for the current time step has ended.

[0037] Furthermore, the current height Calculated based on the pen tip's three-dimensional coordinates, three-dimensional center point, and normal vector.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] This invention provides a pen touch recognition method based on gesture recognition and inertial sensors. Based on a gesture-pen fusion positioning system, it acquires data from an inertial sensor and processes it into the pen tip's three-dimensional spatial coordinates. This data is then processed into the height information of the pen tip from the writing surface. Finally, the touch recognition result is obtained by linking this information with vibration information from the inertial sensor. This invention utilizes the gesture tracking capabilities commonly found in VR / MR / AR devices, avoiding the need for complete pen customization. It can be embedded into existing devices at a lower cost and is not limited to writing surfaces or pen types, exhibiting high versatility. By using pen tip height to assist vibration intensity in touch recognition, the recognition results are made more accurate and reliable. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the overall process of a pen touch recognition method based on gesture recognition and inertial sensors provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the writing pen motion model provided in an embodiment of the present invention. Detailed Implementation

[0043] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0044] Reference Figure 1 This embodiment provides a pen touch recognition method based on gesture recognition and inertial sensors, offering a low-cost and high-precision pen touch recognition solution for virtual reality / augmented reality / mixed reality devices on the market. The method is applied to a gesture-pen fusion positioning system, which includes an inertial sensing device, a pen, a fixing strap, and a host computer.

[0045] The inertial sensor is used to collect the acceleration and attitude information of the writing pen, and transmits the acceleration and attitude information to the host via wireless communication. The inertial sensor can be fixed to the tail of the writing pen by the fixing strap, and the direction of the inertial sensor is aligned with the direction of the pen barrel.

[0046] In one possible implementation, the inertial sensing device includes a microprocessor, a MEMS inertial sensor, a wireless communication module, a battery power supply, and physical buttons. When the user presses the physical button, the physical button sends a trigger signal to the microprocessor. After receiving the trigger signal, the microprocessor extracts the acceleration from the MEMS inertial sensor and the attitude information obtained based on the attitude calculation algorithm. The acceleration and attitude information are then uploaded to the host computer via the wireless communication module. The battery power supply provides the energy required for the operation of the inertial sensing device.

[0047] The writing pen and the securing strap are objects without any electronic components. For example, the writing pen can be any straight, rod-shaped object suitable for a user to hold and write with. The securing strap is used to attach the inertial sensing device to the writing pen.

[0048] The host device is a computing device with computing power, positioning capabilities, wireless communication, and gesture recognition capabilities, such as various virtual reality / augmented reality / mixed reality head-mounted displays. The host device receives information from the inertial sensing device via wireless communication and collects gesture recognition data and its own positioning data to complete the touch recognition of the writing pen.

[0049] Based on the above gesture-pen fusion positioning system, the method provided in this embodiment includes the following steps:

[0050] S101. The acceleration and attitude information of the writing pen are collected by the inertial sensor and uploaded to the host.

[0051] In this embodiment, when the user presses the physical button on the inertial sensor, the data acquisition and wireless communication functions of the inertial sensor are triggered. The inertial sensor then begins to periodically send acceleration and attitude information to the host computer.

[0052] S102. The host computer recognizes the writer's gestures, obtains gesture recognition data, and collects its own positioning data.

[0053] S103. The host calculates the three-dimensional coordinates of the pen tip by fusing positioning based on acceleration and attitude information, gesture recognition data, and positioning data.

[0054] S104. Calculate the vibration intensity of the acceleration obtained by the inertial sensing device.

[0055] S105. Determine whether the writing plane initialization has been performed. If not, proceed to step S106. If yes, proceed to step S107.

[0056] S106. Determine a three-dimensional center point on the writing plane and the normal vector of the writing plane to complete the initialization of the writing plane.

[0057] S107. Calculate the height of the pen tip from the writing plane based on the three-dimensional center point and normal vector, and perform joint touch recognition based on vibration intensity and pen tip height until the system stops.

[0058] In this embodiment, sensor data is first acquired and processed into the positioning coordinates of the pen tip in three-dimensional space, further processed into the height information of the pen tip from the writing plane, and finally combined with the vibration information of the inertial sensing device to obtain the touch recognition result.

[0059] In a preferred embodiment, in step S103, the host computer calculates the three-dimensional coordinates of the writing pen tip based on acceleration and attitude information, gesture recognition data, and positioning data through fusion positioning. This specifically includes the following operations:

[0060] S201. Based on the vectors pointing from the inertial sensor to the grip point at different time points, and the sampling interval of the inertial sensor, a motion model is established, wherein the grip point is the position where the writer holds the pen body.

[0061] S202. Using the motion model as a framework, Kalman fusion localization is performed to obtain the grip point displacement.

[0062] S203. Perform vector calculation based on the displacement of the grip point to obtain the three-dimensional coordinates of the pen tip.

[0063] In this implementation, the relationships between the motion model variables are as follows: Figure 2 As shown, the motion model is represented as follows:

[0064]

[0065] In the above formula, express The displacement of the gripping point at any given moment. express t The speed of holding the point at any moment, express The acceleration of the inertial sensor at any given moment. express t The vector pointing from the inertial sensor to the gripping point at all times. This indicates the sampling interval of the inertial sensing device. express time right t The estimated value of the first derivative, express time and time The difference, express The three-dimensional coordinates of the pen tip at any given moment. express t The vector that points from the grip point to the pen tip at all times.

[0066] As another preferred implementation, the vibration intensity of the acceleration obtained by the inertial sensing device is calculated, specifically: let the acceleration obtained at a certain time point be... The vibration intensity is the vector magnitude processed by the high-pass filter, expressed as:

[0067]

[0068] In the above formula, For acceleration is Vibration intensity at that time Indicates the cutoff frequency as A high-pass filter, for example, with a cutoff frequency of 100Hz.

[0069] As another possible implementation, step S106 specifically includes the following steps:

[0070] S301. Determine several three-dimensional points on the writing plane. , forming a point set .

[0071] In this embodiment, for Matrix. Determining 3D points. The methods include, but are not limited to:

[0072] ① Touch recognition using vibration intensity: Given a fixed vibration intensity threshold, when A touch action is considered detected. A trajectory is drawn on the writing surface with a writing pen, and the points where recognition was successful are recorded.

[0073] ② Use the host computer to scan the writing plane and obtain a series of points on the writing plane; etc.

[0074] S302, Calculate the three-dimensional center point of the point set. The corresponding calculation formula is expressed as:

[0075]

[0076] S303, Based on point set With the three-dimensional center point Calculate the centralized point set With covariance matrix The calculation formula is as follows:

[0077]

[0078] S304. Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and take the eigenvector corresponding to the smallest eigenvalue. .

[0079] S305. Let the coordinates of the writer's head be... Calculate the eigenvector and the vector The dot product.

[0080] S306. Calculate the normal vector of the eigenvector based on the value of the dot product. Output the three-dimensional center point and normal vector .

[0081] In this implementation, if the dot product is less than 0, then the normal vector... Otherwise, normal vector . and This refers to the three-dimensional center point and normal vector on the writing plane to be determined.

[0082] As another preferred embodiment, step S107 specifically includes the following operations:

[0083] S401. Initialize buffers and variables , The buffer is used to record the three-dimensional coordinates of the pen tip in the state to be determined. Used to record the height of the pen tip at the moment of contact with the writing surface. Used to record flag bits.

[0084] S402. Based on the vibration intensity, perform touch recognition to determine whether the writing pen touches the writing surface. If the recognition result is touch, proceed to step S403; otherwise, proceed to step S404.

[0085] In this embodiment, a fixed vibration intensity threshold is given. ,like If so, it is considered that a touch action has been detected.

[0086] S403. Mark the current three-dimensional coordinates of the pen tip as a touch, flag bit. Set to true. Update to current height Mark all pen tip 3D coordinates in the buffer as touched and clear the buffer, then proceed to step S406.

[0087] In this implementation, the current height Based on the three-dimensional coordinates of the pen tip 3D center point and normal vector The calculation is performed, and the corresponding formula is expressed as follows:

[0088] h

[0089] S404, Determine the flag bit If the value is true, proceed to step S405; otherwise, mark the current pen tip's three-dimensional coordinates as non-touch and proceed to step S406.

[0090] S405. Calculate the height difference between the current height and the height at the last touch moment. If the height difference Less than the preset height difference threshold If the three-dimensional coordinates of the pen tip are correct, then put them into the buffer; otherwise, set the flag. Set to false, mark all pen tip 3D coordinates in the buffer as non-touched, and clear the buffer.

[0091] S406, Touch recognition for the current time step has ended.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gesture recognition and inertial sensor based writing pen touch recognition method, characterized by, The method is applied to a gesture-pen fusion positioning system, and comprises the following steps: S101, collecting acceleration and posture information of a writing pen by an inertial sensing device, and uploading the acceleration and posture information to a host computer; S102, identifying a hand gesture of a writer by the host computer, obtaining hand gesture identification data, and collecting positioning data of the host computer itself; S103, calculating three-dimensional coordinates of a pen tip of the writing pen by the host computer according to the acceleration and posture information, the hand gesture identification data, and the positioning data through fusion positioning; S104, calculate the vibration intensity of the acceleration obtained by the inertial sensing device, specifically: let the acceleration collected by the inertial sensor device be Then the vibration intensity is the vector module length processed by the high-pass filter, which is represented as: In the above formula, is the vibration intensity when the acceleration is is a high-pass filter with a cutoff frequency of ;​ S105, judging whether writing plane initialization has been performed, and if not, performing step S106, and if yes, performing step S107; S106, determining a three-dimensional center point on a writing plane and a normal vector of the writing plane, and completing writing plane initialization; S107, calculating a height of the pen tip of the writing pen to the writing plane according to the three-dimensional center point and the normal vector, and performing joint touch recognition based on vibration intensity and the height of the pen tip until the system stops, and specifically comprising the following operations: S401, initialize buffer and variables , buffer for recording the 3D coordinates of the stylus tip to be determined, buffer for recording the height of the stylus tip at the moment of touching the writing plane, buffer for recording the flag bit; S402, performing touch recognition on whether the writing pen touches the writing plane according to the vibration intensity, and if the recognition result is touch, performing step S403, and if not, performing step S404; S403. Mark the current three-dimensional coordinates of the pen tip as a touch, flag bit. Set to true. Update to current height Mark all three-dimensional coordinates of the pen tip in the buffer as touched and clear the buffer, then jump to step S406; S404, judging the flag bit whether true, if yes, jump to step S405, otherwise mark the pen tip three-dimensional coordinate at the current time as non-touch, and jump to step S406; S405, calculate the height difference between the current height and the height at the last touch time, if the height difference is less than a preset height difference threshold, put the pen tip three-dimensional coordinates into the buffer, otherwise set the flag bit to false, mark all pen tip three-dimensional coordinates in the buffer as non-touch, and empty the buffer. S405, calculate the height difference between the current height and the height at the last touch time, if the height difference is less than a preset height difference threshold, put the pen tip three-dimensional coordinates into the buffer, otherwise set the flag bit to false, mark all pen tip three-dimensional coordinates in the buffer as non-touch, and empty the buffer. S406, ending touch recognition of the current time step.

2. The method according to claim 1, wherein the method is characterized by, The gesture-pen fusion positioning system comprises an inertial sensing device, a writing pen, a fixed band, and a host computer, the inertial sensing device is used for collecting acceleration and posture information of the writing pen, and transmitting the acceleration and posture information to the host computer through wireless communication, the inertial sensing device is fixed at a tail portion of the writing pen through the fixed band, and the direction of the inertial sensing device is aligned with the direction of a pen shaft, and the host computer is a computing device with computing capability, positioning capability, wireless communication capability, and gesture recognition capability. 3.The writing pen touch recognition method based on gesture recognition and inertial sensor according to claim 2, wherein, The inertial sensing device comprises a microprocessor, a MEMS inertial sensor, a wireless communication module, a battery power supply, and a physical button, the physical button is used for sending a trigger signal to the microprocessor when being pressed, the microprocessor extracts acceleration of the MEMS inertial sensor and posture information obtained based on a posture calculation algorithm after receiving the trigger signal, and uploads the acceleration and the posture information to the host computer through the wireless communication module, and the battery power supply is used for providing energy required for running of the inertial sensing device. 4.The writing pen touch recognition method based on gesture recognition and inertial sensor according to claim 1, wherein, Step S103 specifically comprises the following operations: S201, establishing a motion model based on vectors from the inertial sensing device to a holding point at different time nodes and sampling interval time of the inertial sensing device, the holding point being a position at which a writer holds a pen shaft of the writing pen; S202, performing Kalman filter fusion positioning in the framework of the motion model to obtain displacement of the holding point; S203, performing vector calculation according to the displacement of the holding point to obtain three-dimensional coordinates of a pen tip. 5.The writing pen touch recognition method based on gesture recognition and inertial sensor according to claim 1, wherein, Step S106 specifically comprises the following operations: S301, determining three-dimensional points on a number of writing planes , constituting a point set ; S302、calculating a three-dimensional center point on the point set ; S303、according to the point set with the three-dimensional center point , calculate the centralized point set and the covariance matrix; S304, calculating eigenvalues and eigenvectors of a covariance matrix, and taking an eigenvector corresponding to a minimum eigenvalue; S305, set the writer head coordinate as , calculate the dot product of the feature vector and the vector ; S306, calculating a normal vector of the eigenvector according to a value of a dot product, and outputting a three-dimensional center point and the normal vector. 6.The gesture recognition and inertial sensor based stylus touch recognition method of claim 5, wherein, Current height According to the three-dimensional coordinates of the pen tip, the three-dimensional center point, and the normal vector.

Images