Intelligent pen control method, head-mounted display device and storage medium

Abstract

The invention discloses a control method of an intelligent pen, head-mounted display equipment and a storage medium, relates to the technical field of electric data processing, and discloses the control method of the intelligent pen, which is applied to the head-mounted display equipment, the head-mounted display equipment is connected with the intelligent pen, and a vibration motor is arranged at a pen point of the intelligent pen. The control of the intelligent pen comprises the following steps: detecting that a pen point is in contact with a writing plane of a virtual scene of the head-mounted display equipment, and acquiring a contact distance and a moving direction between the pen point and the writing plane; determining vibration parameters of the vibration motor according to the contact distance and the moving direction; and controlling the vibration motor to operate according to the vibration parameter, so that the intelligent pen generates vibration feedback during writing in the virtual scene. On the basis, when the user uses the intelligent pen to write in the virtual scene, the contact state and the movement change with the virtual writing plane are sensed through the vibration of the pen point, and the touch experience in the virtual writing process is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric data processing, and particularly relates to a control method of a smart pen, a head-mounted display device and a storage medium. BACKGROUND

[0002] In an XR (Extended Reality) device such as an XR head-mounted display, smart glasses, etc., such as AR (Augmented Reality), VR (Virtual Reality), MR (MixedReality), a user can see virtual objects or scenes that cannot be directly observed by the naked eye.

[0003] The smart pen is a cross-platform general interaction device and is widely used in traditional display terminals such as tablets and computers. The smart pen can also be used as a supplementary interaction tool for an XR scene to help the user interact more accurately with virtual content displayed in the XR device.

[0004] However, when the smart pen is used as a supplementary interaction tool for the XR device to realize air writing in a virtual scene, the smart pen can only draw content, and the user cannot perceive the details of the writing action, which reduces the interaction experience of the virtual scene.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a control method of a smart pen, a head-mounted display device and a storage medium, which aims to solve the technical problem that the interaction of a virtual scene is poor when writing in the virtual scene because of the lack of tactile feedback in the real writing process.

[0007] To achieve the above purpose, the present application provides a control method of a smart pen, which is applied to a head-mounted display device, the head-mounted display device is connected with the smart pen, a vibration motor is arranged at a pen tip of the smart pen, and the control method of the smart pen comprises the following steps: detecting that the pen tip contacts a writing plane of a virtual scene of the head-mounted display device, acquiring a contact distance between the pen tip and the writing plane and a moving direction; determining vibration parameters of the vibration motor according to the contact distance and the moving direction; controlling the vibration motor to operate according to the vibration parameters, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0008] In an embodiment, the vibration motor comprises a horizontal vibration motor and a vertical vibration motor, the vibration parameter comprises a horizontal vibration parameter and a vertical vibration parameter, and the step of determining the vibration parameter of the vibration motor according to the contact distance and the moving direction comprises: calculating the vertical vibration parameter of the vertical vibration motor according to the absolute value of the contact distance, the absolute value being positively correlated with the amplitude of the vertical vibration parameter when the pen nib and the writing plane are in contact; determining the moving speed of the pen nib when moving based on the moving direction; determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed.

[0009] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed, the control method of the smart pen comprises: obtaining the inclination angle between the smart pen and the writing plane; determining a first horizontal amplitude corresponding to the inclination angle, and determining a second horizontal amplitude corresponding to the moving speed; weighting and fusing the first horizontal amplitude and the second horizontal amplitude to obtain a target amplitude, and updating the current amplitude of the horizontal vibration parameter based on the target amplitude.

[0010] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed, the control method of the smart pen further comprises: determining a turning time corresponding to a turning point when the moving direction of the pen nib changes; updating a third horizontal amplitude corresponding to the turning time based on a preset turning vibration parameter, and rotating the horizontal vibration motor based on the third horizontal amplitude in a time period corresponding to the turning time.

[0011] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed, the control method of the smart pen further comprises: calculating a horizontal vibration component of the vertical vibration parameter in the horizontal direction. superimpose the horizontal vibration component in the horizontal vibration parameter to obtain a target horizontal vibration parameter capable of canceling the horizontal vibration component; based on the target horizontal vibration parameter and the vertical vibration parameter, control the vibration motor to operate to make the smart pen generate vibration feedback when writing in the virtual scene.

[0012] In an embodiment, after the step of controlling the vibration motor to operate according to the vibration parameter to make the smart pen generate vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises: determining a vibration amplitude value in the vibration parameter perpendicular to the writing plane; determining a target developing parameter of the text to be rendered according to a correspondence between the vibration amplitude value and the developing parameter; rendering the writing content of the smart pen based on the target developing parameter.

[0013] In an embodiment, after the step of controlling the vibration motor to operate according to the vibration parameter to make the smart pen generate vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises: obtaining an audio playing parameter corresponding to the vibration amplitude value of the vibration parameter; outputting a writing sound effect when the smart pen writes based on the audio playing parameter.

[0014] In an embodiment, the control method of the smart pen further comprises: in response to an erasing instruction, entering an erasing mode and obtaining an erasing distance of the pen tip when the pen tip contacts the writing plane; determining an erasing vibration parameter of the vibration motor according to the erasing distance; controlling the vibration motor to operate according to the erasing vibration parameter to make the smart pen generate erasing vibration feedback.

[0015] In addition, to achieve the above-mentioned purposes, the present application further proposes a head-mounted display device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the control method of the smart pen as described above.

[0016] In addition, to achieve the above-mentioned purposes, the present application further proposes a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the control method of the smart pen as described above.

[0017] The one or more technical solutions provided in the application have at least the following technical effects: The head-mounted display device is connected to the smart pen, a vibration motor is arranged at the tip of the smart pen, so that when the tip is detected to be in contact with the writing plane of the virtual scene of the head-mounted display device, the contact distance and the moving direction between the tip and the writing plane are obtained, and then the vibration parameters of the vibration motor are determined according to the obtained information and the vibration motor is controlled to operate to generate vibration feedback, so that when the user uses the smart pen to write in the virtual scene, the vibration of the tip is used to enhance the touch experience in the virtual writing process, improve the accurate perception and immersion of the user to the operation of the smart pen, make the writing operation in the virtual scene closer to the real writing feeling, and improve the interactive experience when writing in the virtual scene. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present application and serve to explain the principles of the application, together with the description.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0020] Figure 1 The flowchart provided for the first embodiment of the control method of the smart pen of the present application; Figure 2 The flowchart provided for the second embodiment of the control method of the smart pen of the present application; Figure 3 The flowchart provided for the fifth embodiment of the control method of the smart pen of the present application; Figure 4 The device structure schematic diagram of the hardware running environment involved in the control method of the smart pen in the embodiments of the present application.

[0021] The purpose of the present application, the functional characteristics and the advantages will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0023] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail with reference to the drawings and the specific embodiments.

[0024] In XR devices such as XR headsets, smart glasses, and the like, users can see virtual objects or scenes that cannot be directly observed by the naked eye. As a cross-platform universal interaction device, a smart pen is widely used in traditional display terminals such as tablets and computers, and can also be used as a supplementary interaction tool for XR scenes to help users interact more accurately with virtual content displayed in the XR device.

[0025] However, as a supplementary interaction tool for XR devices, the smart pen can only draw content when used for air writing in a virtual scene, and users cannot perceive the details of the writing action, which reduces the interactive experience of the virtual scene.

[0026] Based on this, the main solution of the embodiments of the present application is to detect that the pen tip contacts a writing plane of a virtual scene of the head-mounted display device, obtain a contact distance between the pen tip and the writing plane and a moving direction; According to the contact distance and the moving direction, the vibration parameters of the vibration motor are determined; According to the vibration parameters, the vibration motor is controlled to operate, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0027] Specifically, by setting a vibration motor on the pen tip of the smart pen, when the pen tip contacts the writing plane of the virtual scene of the head-mounted display device, the contact distance between the pen tip and the writing plane and the moving direction are obtained, and then the vibration parameters of the vibration motor are determined according to the obtained information and the vibration motor is controlled to operate to generate vibration feedback, so that when the user uses the smart pen to write in the virtual scene, the vibration of the pen tip enhances the tactile experience in the virtual writing process.

[0028] It should be noted that the execution subject of the present embodiment can be a head-mounted display device having data processing, network communication, and program running functions, etc. The head-mounted display device includes VR glasses, AR glasses, MR glasses, etc. The present embodiment and the following embodiments will be described below with the head-mounted display device as an example.

[0029] Based on this, the embodiments of the present application provide a control method of a smart pen, applied to a head-mounted display device, the head-mounted display device being connected with the smart pen, and a user writing in a virtual scene through the smart pen. The pen tip of the smart pen is provided with a vibration motor, so as to provide a tactile experience in the virtual scene through the vibration motor when the user writes.

[0030] Please refer to Figure 1 , Figure 1 The flowchart of the first embodiment of the control method of the smart pen of the present application is shown in the figure.

[0031] In the present embodiment, the control method of the smart pen includes steps S10-S30: In step S10, when it is detected that the pen tip is in contact with the writing plane of the virtual scene of the head-mounted display device, the contact distance between the pen tip and the writing plane and the moving direction are obtained.

[0032] In this embodiment, the head-mounted display device tracks the pen tip of the smart pen by infrared visual positioning technology, pure visual positioning technology, or similar VR handle position tracking technology, so as to detect whether the pen tip of the smart pen falls in the virtual plane of the virtual scene that needs to be written, that is, to determine whether the pen tip is in contact with the writing plane. The writing plane is a plane in the virtual scene that needs to be written on the surface of a virtual object. The contact distance is the vertical spatial distance from the pen tip to the virtual writing plane, that is, the vertical contact distance in the direction perpendicular to the writing plane; the moving direction is the motion trajectory vector of the pen tip on the writing plane of the virtual scene, including the direction angle and the motion speed.

[0033] As an optional implementation, after the user wears the head-mounted display device and uses the smart pen, the head-mounted display device captures the infrared signal of the pen tip in real time based on the infrared camera, and converts the captured signal into a spatial coordinate. Meanwhile, the head-mounted display device can determine the three-dimensional coordinates of the writing plane based on the virtual scene constructed by itself. Then, when the user moves the pen tip close to the position of the virtual plane, the coordinates of the pen tip gradually approach the coordinates of the writing plane. When the distance between the coordinates of the pen tip and the writing plane is less than a preset threshold, such as less than or equal to 0, it is determined that the pen tip is in contact with the writing plane. In the contact state, the head-mounted display device continuously records the change of the coordinates of the pen tip, and calculates the displacement direction and the moving speed in a unit time through the difference between adjacent coordinate points. It can be understood that when the user writes, the user will lift the hand after writing a character, and at this time, the pen tip is not in contact with the writing plane. Therefore, after detecting that the pen tip is in contact with the writing plane, the head-mounted display device considers that the user is in a writing state, and continuously collects the contact distance and the moving direction of the pen tip and the writing plane on the premise that the pen tip is not obviously lifted, such as the distance between the pen tip and the writing plane being within a preset distance, such as less than 0.1 meters. The distance between the coordinates of the pen tip and the coordinates of the writing plane can be less than 0, that is, to ensure that the user writes on the virtual writing plane.

[0034] For example, when the user writes characters on a virtual blackboard using the smart pen, the infrared camera of the smart glasses captures the coordinates of the infrared point of the pen tip, and calculates that the vertical distance from the pen tip to the blackboard plane is -0.6 mm. At this time, it is determined that the pen tip is in contact with the blackboard plane, and it is determined that the pen tip moves to the right and downward at a speed of 2.5 cm / s through the continuous change of the coordinates.

[0035] It should be noted that the above parameters are only used for explanation and illustration, and are not limited to the present application.

[0036] Step S20, determining a vibration parameter of the vibration motor according to the contact distance and the moving direction.

[0037] In the embodiment, the vibration parameter is a core parameter for controlling the operation of the vibration motor, including vibration frequency, amplitude and vibration direction, etc. Based on the vibration parameter, the vibration motor feeds back force parallel to the writing plane to the user through vibration. Alternatively, in addition to the force parallel to the writing plane, the normal writing also includes the force perpendicular to the writing plane, so the vibration motor can also feed back the force parallel to the writing plane and the force perpendicular to the writing plane to the user through vibration. Among them, different directions of force can be realized by one vibration motor provided at the pen tip, or different directions of force can be provided by multiple vibration motors provided at the pen tip, such as two vibration motors. When the two vibration motors operate based on the vibration parameter, they respectively provide forces in parallel and perpendicular directions.

[0038] Further, the vibration parameter can be calculated based on the contact distance and the moving direction through a preset mapping relationship, an algorithm model, etc. Specifically, the head-mounted display device calls a pre-stored mapping table, which is established based on a large amount of experimental data and contains the corresponding relationship between the contact distance and the amplitude and the corresponding relationship between the moving direction change rate and the vibration frequency. The obtained contact distance is input into the mapping table, so as to match the corresponding vibration amplitude. The calculated moving direction is input into the mapping table, so as to match the corresponding vibration frequency. Based on this, the vibration parameter of the vibration motor is obtained.

[0039] Alternatively, the head-mounted display device can use a dynamic adjustment algorithm based on a neural network, take the contact distance and the moving direction change rate as input features, and then use a model trained by historical writing data to calculate the vibration amplitude matched with the current interaction state in real time, such as reducing the distance to reduce the amplitude and changing the direction to increase the vibration frequency, so as to obtain the corresponding vibration parameter.

[0040] It can be understood that when there are multiple vibration motors, the vibration parameter corresponding to each vibration motor needs to be calculated, so that the vibration motor can provide forces in different directions when vibrating.

[0041] Step S30, controlling the vibration motor to operate according to the vibration parameter, so as to make the smart pen generate vibration feedback when writing in the virtual scene.

[0042] In the embodiment, after the head-mounted display device determines the vibration parameter, it sends it to the smart pen, so that the vibration motor in the pen tip rotates, thereby generating vibration feedback when writing in the virtual scene, including friction force opposite to the moving direction, or friction force opposite to the moving direction and rebound force perpendicular to the writing plane, so that the user perceives the writing resistance when writing, or the writing resistance and the force of the pen tip pressing the paper.

[0043] It should be noted that the head-mounted display device starts to calculate the real-time vibration parameter when the pen tip contacts the writing plane of the virtual scene, so that the user can feel the physical feedback caused by the vibration of the pen tip in the virtual scene when writing with the pen tip.

[0044] When the pen tip is lifted and leaves the writing plane, the vibration motor of the pen tip stops vibrating. When the pen tip is in a static state, the vibration motor also stops vibrating.

[0045] Optionally, in addition to determining the vibration parameter through the head-mounted display device, so as to control the vibration motor to operate to generate the vibration feedback when writing based on the vibration parameter, the smart pen also sets a vibration mode, and sets a corresponding vibration mode based on the writing habit of the user, so that the selected vibration mode vibrates when the pen tip contacts the writing plane.

[0046] Optionally, simple vibration feedback can also be performed through the pen tip, such as controlling the pen tip to vibrate to control the vibration motor to operate when the active range of the pen tip is small.

[0047] The embodiment provides a control method of a smart pen. The pen tip contact distance and movement direction when the pen tip contacts a writing plane in a virtual scene are detected through a head-mounted display device. Parameters required for the vibration motor to vibrate are calculated according to the contact distance and movement direction. The vibration motor at the pen tip is controlled to operate based on the parameters, vibration feedback of writing operation in the virtual scene is realized, and the sense of reality and immersion of virtual writing is improved, so that the user can obtain a consistent tactile experience as real writing in the virtual scene.

[0048] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, the vibration motor includes a horizontal vibration motor and a vertical vibration motor, and the vibration parameter includes a horizontal vibration parameter and a vertical vibration parameter. The writing resistance is generated through the horizontal vibration motor, and the pressure of the pen tip on the paper is generated through the vertical vibration motor. Based on this, please refer to Figure 2 , step S20 further includes steps S21-S23: Step S21, according to the absolute value of the contact distance, the vertical vibration parameter of the vertical vibration motor is calculated.

[0049] In the embodiment, when the pen tip contacts the writing plane, the absolute value of the contact distance and the amplitude of the vertical vibration parameter are positively correlated. For example, the absolute value of the contact distance increases by A millimeters, and the vertical vibration amplitude increases by B millimeters, or the contact distance is in the interval a1-a2, and the corresponding vibration amplitude is b1 millimeters, and in the interval a3-a4, the corresponding vibration amplitude is b2 millimeters, wherein a3>a2, and b2 is greater than b2.

[0050] In other words, when the pen tip penetrates the writing plane, the vertical vibration intensity will become stronger, i.e., the absolute value of the amplitude and the contact distance are proportional. Therefore, the amplitude of the vertical vibration can be calculated through the mapping relationship between the contact distance and the amplitude. For example, the amplitude = k * contact distance, and k is a mapping parameter. In addition, the vibration frequency can also be calculated through the mapping relationship between the contact distance and the vibration frequency. The specific mapping method is not limited herein.

[0051] It should be noted that the greater the amplitude, the stronger the impact of the vibration motor on the shell, and the greater the resistance that the user can feel caused by the vibration.

[0052] Step S22, determining the moving speed of the pen tip based on the moving direction.

[0053] After obtaining the vertical vibration parameters of the vertical vibration motor, the horizontal vibration parameters, including the vibration amplitude and the vibration direction, need to be calculated to generate the writing resistance opposite to the writing direction.

[0054] Therefore, the head-mounted display device needs to determine the moving speed of the pen tip based on the moving direction through the coordinate change speed of the pen tip.

[0055] Step S23, determining the horizontal vibration parameters of the horizontal vibration motor according to the moving direction and the moving speed.

[0056] In this embodiment, the direction of the horizontal vibration method parameters is opposite to the moving direction, and the amplitude of the horizontal vibration method parameters is negatively correlated with the moving speed. Therefore, the amplitude corresponding to the moving speed can also be determined through the mapping relationship. In the normal moving process, when the pen tip contacts and moves relative to the writing plane, the vibration intensity is the strongest at the moment of moving, and the vibration intensity decreases after moving and keeps a slight vibration, thereby simulating the paper friction characteristics and generating the writing resistance. When the pen tip moves quickly on a real plane such as paper, the contact time of the pen tip with the surface per unit area is shortened, and the continuous perception intensity of the friction force is reduced, i.e., the existence of the friction resistance felt by the finger when quickly drawing on the paper is weaker than the resistance felt when slowly writing. Therefore, the moving speed and the amplitude of the horizontal vibration parameters are negatively correlated in the virtual scene, so that the physical characteristics of the real writing can be accurately simulated in the virtual scene, the reality of the virtual writing is enhanced, and the interaction experience is improved.

[0057] It should be noted that the horizontal vibration motor generates writing resistance in the writing plane when vibrating, and thus the vibration direction can be any direction in the plane. Therefore, in order to avoid the resistance direction not corresponding to the actual writing direction, the vibration direction of the horizontal vibration motor needs to be determined by the moving direction, such as moving to the right, the vibration direction is to the left, so as to generate writing resistance in the opposite direction. Similarly, in the vertical vibration parameter of the vertical vibration motor, the vibration direction is opposite to the direction of the pen tip of the smart pen.

[0058] The embodiment provides a control method of a smart pen. By arranging a horizontal vibration motor and a vertical vibration motor, vibration control in different directions is respectively realized based on the two different vibration motors, so as to generate paper resistance when writing and rebound resistance of the pen tip pressing the paper. Meanwhile, the moving speed in the horizontal direction and the vibration amplitude are inversely proportional, so as to simulate the case that the friction is smaller when the pen tip slides fast in real writing, so that the user perceives the writing action details in the virtual scene, and the interactive experience when writing in the virtual scene is improved.

[0059] Based on the second embodiment of the present application, in the third embodiment of the present application, when writing in the simulation scene, different tilt angles correspond to different horizontal vibration intensities. For example, when the pen tip is close to vertical writing, the pressure of the pen is mainly concentrated on the tip contact point of the pen tip, the normal pressure of the pen tip on the paper is larger, the friction resistance of the paper fiber on the pen tip is more sufficient, and the finger can clearly feel the resistance feedback. When the pen tip is greatly inclined, the pressure of the pen is dispersed to the side contact area of the pen tip, the effective normal pressure is reduced, the friction resistance of the pen tip on the paper naturally weakens, and the resistance feeling felt by the finger is obviously weakened. Therefore, in this embodiment, when calculating the horizontal vibration parameter of the horizontal vibration motor, the current tilt angle of the pen tip also needs to be analyzed and calculated. Therefore, after step S23, steps S24-S26 are further included. Step S24, acquiring a tilt angle between the smart pen and a writing plane.

[0060] Step S25, determining a first horizontal amplitude corresponding to the tilt angle, and determining a second horizontal amplitude corresponding to the moving speed.

[0061] Step S26, weighting and fusing the first horizontal amplitude and the second horizontal amplitude to obtain a target amplitude, and updating a current amplitude of the horizontal vibration parameter based on the target amplitude.

[0062] In the embodiment, the head-mounted display device can determine the tilt angle between the smart pen and the writing plane through image analysis. Then, based on the preset mapping relationship, the first horizontal amplitude corresponding to the tilt angle is calculated. It can be understood that the horizontal amplitude refers to the amplitude of the horizontal vibration motor vibration. Similarly, based on the mapping relationship between the moving speed and the amplitude, the second horizontal amplitude is calculated. Finally, the target amplitude corresponding to the current moving speed and the tilt angle at the time of writing is obtained by weighting and fusing the two amplitudes, and the current amplitude in the horizontal vibration parameter is updated based on the target amplitude, so as to improve the reality of the horizontal vibration motor vibration in the virtual scene and improve the user's interaction experience. The calculation process based on the mapping relationship is similar to the calculation process of the foregoing embodiment, and will not be described herein. The amplitude of the current amplitude is calculated in real time.

[0063] For example, the amplitude calculated based on the moving speed is C1 millimeters. Under normal circumstances, the horizontal vibration motor vibrates based on the amplitude of C1 millimeters. Since there is a certain tilt between the smart pen and the writing plane, the amplitude calculated based on the tilt angle is C2 millimeters. The target amplitude C=aC1+b*C2 is obtained by weighting and summing the two amplitudes, and finally the horizontal vibration motor vibrates based on the target amplitude C calculated in real time.

[0064] The amplitude of the horizontal vibration is adjusted by the tilt angle of the smart pen, so that the vibration is more consistent with the force scenario during real writing, and the interaction experience during writing in the virtual scene is improved.

[0065] Based on the second embodiment or the third embodiment of the present application, in the fourth embodiment of the present application, in addition to the tilt angle affecting the real vibration feedback, in the actual writing process, the pen tip slides uniformly in a fixed direction, the friction between the pen tip and the paper surface is stable, the resistance perceived by the hand is smooth and continuous, and when the pen tip turns, the pen tip needs to change the moving direction. At this time, the contact point between the pen tip and the paper surface is instantaneously extruded due to the change of direction, the normal pressure becomes larger, and the resistance of the paper fiber to the pen tip is enhanced. In this process, the sliding friction is converted into micro-jitter friction, and the resistance reaches a peak value in a very short time. After the turning, the pen tip moves towards the new moving direction, and at this time the friction force stabilizes and quickly falls back to a new stable value along the direction.

[0066] Therefore, to improve the vibration feedback effect of the virtual scene, after step S23, the method further comprises: determining a turning point corresponding to a moment when the moving direction of the pen tip changes, and then updating a third horizontal amplitude corresponding to the turning point based on a preset turning vibration parameter. The head-mounted display device pre-stores the preset turning vibration parameter corresponding to the turning point, in other words, each turning point corresponds to a set of vibration parameters, for example, the amplitude first increases and then decreases. Based on the preset turning vibration parameter, the third horizontal amplitude corresponding to the turning point is updated, so that the horizontal vibration motor rotates based on the third horizontal amplitude at the period corresponding to the turning point, and then rotates based on the amplitude calculated based on the real-time moving speed and the inclination angle.

[0067] By setting the corresponding vibration parameter for the turning point of the pen tip, the vibration feedback effect of the user when writing in the virtual scene is further improved, and the interactive experience when writing in the virtual scene is improved.

[0068] Based on the second embodiment, the third embodiment or the fourth embodiment of the present application, in the fifth embodiment of the present application, according to the usage habit of normal people, when the user writes in the virtual scene, the smart pen generally has a certain inclination angle with the writing plane. At this time, the force generated by the vibration perpendicular to the writing direction includes two parts, one part is the feedback force simulated by the pen tip pressing downward vertically, and the other part is the horizontal force parallel to the writing plane.

[0069] When the smart pen is inclined, the vibration of the vertical vibration motor will be angle-decomposed to generate an additional horizontal component force. This horizontal component force is neither generated by the user's active pressing nor required by the simulated sliding resistance. This resistance will make the user perceive a strange lateral force, for example, the user only presses the pen tip downward, but feels that the pen tip is slightly pulled to the left or right. This touch feeling is completely inconsistent with real writing, which further breaks the realism of the virtual scene and reduces the interactive experience.

[0070] Therefore, in the present embodiment, please refer to Figure 3 , after step S23, the method further comprises steps S40-S60: Step S40, calculating a horizontal vibration component of the vertical vibration parameter in the horizontal direction.

[0071] In the present embodiment, the way of calculating the horizontal vibration component of the vertical vibration parameter can be to decompose the force generated by the inclined vertical vibration into a vertical vibration component perpendicular to the writing plane and a horizontal vibration component parallel to the writing plane. For example, the horizontal vibration component is equal to the vertical amplitude multiplied by sinθ when the inclination angle is θ.

[0072] It is understandable that the vibration effect produced by the vertical vibration motor can be considered as the effect on the Z-axis, while the vibration effect produced by the horizontal vibration motor is the effect on the XY plane. When the smart pen is tilted, the vertical vibration motor will generate a component force on the XY plane in addition to the force on the Z-axis.

[0073] Step S50: The horizontal vibration component is superimposed on the horizontal vibration parameter to obtain a target horizontal vibration parameter that can cancel out the horizontal vibration component.

[0074] In this embodiment, the force generated by the horizontal vibration motor needs to satisfy: interference force + horizontal motor force = target friction force, where the interference force is the horizontal vibration component, and the horizontal motor force is the force generated by the horizontal vibration motor itself during vibration. Therefore, the target friction force is the vector sum of the two forces. Thus, the horizontal vibration component is superimposed on the horizontal vibration parameters by adding the forces, thereby canceling out this ineffective force. For example, the amplitude of the horizontal vibration motor when canceling the interference force is calculated using the amplitude formula for simple harmonic motion synthesis, and the phase of the horizontal vibration motor when canceling the interference force is calculated using the phase synthesis formula.

[0075] Step S60: Based on the target horizontal vibration parameters and the vertical vibration parameters, control the vibration motor to operate so that the smart pen generates vibration feedback when writing in the virtual scene.

[0076] After obtaining the target horizontal vibration parameters, based on the final horizontal and vertical vibration parameters, the action of step S30 is executed, that is, the vibration motor is controlled to run. In this way, under the premise that the smart pen is tilted to the writing plane, the tilt interference is eliminated while the interference caused by the vertical vibration motor is reduced.

[0077] For example, the process of canceling the horizontal component of the force in a vertical vibration motor is illustrated with data examples: The current tilt angle of the smart pen is 30°. Calculate the amplitude A of the vertical vibration motor at the current moment. vertical =0.5mm, the horizontal disturbance vibration component generated by the vertical vibration motor is: A disturb =A vertical* sinθ, where the phase φ disturb =0, meaning the vertical vibration is along the pen axis and the horizontal component of the force has a phase of 0. Substituting the data into the formula, we obtain the horizontal disturbance vibration component A. disturb It is 0.25mm.

[0078] In the horizontal vibration parameters, the calculated original horizontal vibration component A horiz =0.4mm, phase is π, opposite to the direction of the disturbance force. Therefore, the target amplitude A of the target horizontal vibration parameter can be obtained by substituting it into the following amplitude formula.target Substitute the above parameters into the formula to obtain A target = 0.15mm. Wherein, φ horiz is the original horizontal phase.

[0079] At the same time, the phase calculation formula is applied to obtain the actual phase of the target horizontal vibration parameter.

[0080] Substitute the above known parameters into the formula to obtain the target phase φ target = π.

[0081] That is, the final target horizontal vibration parameter is obtained, the amplitude is 0.15mm, and the phase is π.

[0082] It should be noted that the above examples and parameters are only used for explanation and are not limited to the calculation method and calculation parameters.

[0083] The embodiment calculates the phase difference of the two motors in real time, thereby calculating the vibration amplitude and phase of the horizontal vibration motor when actually vibrating, so as to offset or fuse the horizontal component of the vertical motor, so that the user perceives the friction force without interference and in accordance with the real physical effect, solves the problem of force interference when writing in the inclined state, and improves the interaction and immersion of virtual writing.

[0084] Based on any one of the above embodiments of the application, in the sixth embodiment of the application, after the vibration motor is controlled to operate based on the vibration parameter to generate vibration feedback, the corresponding picture or output audio can be rendered based on the vibration force generated by the vibration parameter, thereby improving the interaction experience in the virtual scene.

[0085] Specifically, after step S30, the method further includes: determining the vibration amplitude in the vibration parameter perpendicular to the writing plane, then determining the target developing parameter of the text to be rendered according to the corresponding relationship between the vibration amplitude and the developing parameter, and finally rendering the writing content of the smart pen based on the display parameter. The developing parameter can be ink bleeding effect, color depth effect, etc. Taking the ink bleeding effect as an example, when the user writes with the smart pen, the ink bleeding effect can be rendered and displayed in real time. The deeper the vertical writing is into the writing plane, the thicker the stroke thickness is set, and vice versa. Better reflect the actual effect of the user's pen. And the faster the horizontal writing speed is, the more it will produce no ink, similar to the effect of a dry pen. Thus, the interaction experience in the virtual scene.

[0086] ​​​Furthermore, when the smart pen's vibration motor vibrates, the audio playback parameters corresponding to the vibration amplitude of the vibration parameters can be obtained. Then, based on the audio playback parameters, the writing sound effect of the smart pen during writing is output. Simultaneously with writing, the corresponding sound effect is played, such as the swishing sound accompanying the user's writing, enhancing the user experience. The vibration is determined to match the corresponding audio playback parameters through a correspondence, such as a larger vibration amplitude resulting in a louder volume. This correspondence can be a preset relationship, which is not limited herein.

[0087] Furthermore, when the user finishes writing and erases the content, the vibration parameters can be determined based on the parameters during the erasure process, thereby controlling the operation of the vibration motor. For example, vertical vibration can simulate the erasure force, and horizontal vibration can simulate the erasure speed, making the vibration effect consistent with the writing mode. Therefore, after responding to the erase command, the head-mounted display enters the erase mode, then acquires the erasure distance when the pen tip contacts the writing point. The method of acquiring data is the same as the writing process, and will not be elaborated here. Subsequently, the erasure vibration parameters of the vibration motor are determined based on the erasure distance, and finally, the operation of the vibration motor is controlled based on the erasure vibration parameters, thus enabling the smart pen to produce an erasure vibration feedback effect.

[0088] This embodiment enhances the interactive experience in a virtual scene by providing corresponding vibration imaging effects and vibration sound effects while writing. It also sets an erase mode, whose vibration logic is the same as that of the writing mode, further improving the interactive experience of using the smart pen.

[0089] This application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the smart pen control method in the first embodiment described above.

[0090] The following is for reference. Figure 4 It shows a schematic diagram of a structure suitable for implementing the head-mounted display device of the present application embodiments. Figure 4 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0091] like Figure 4As shown, the head-mounted display device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the head-mounted display device are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the head-mounted display device to communicate wirelessly or wired with other devices to exchange data. Although the head-mounted display device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0092] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0093] The head-mounted display device provided by the present application adopts the control method of the smart pen in the above-mentioned embodiments, and can solve the technical problem that the interaction of the virtual scene is poor due to the lack of tactile feedback in the real writing process when writing in the virtual scene. Compared with the prior art, the head-mounted display device provided by the present application has the same beneficial effects as the control method of the smart pen provided by the above-mentioned embodiments, and other technical features in the head-mounted display device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0094] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0095] The above description is merely illustrative of the application and not restrictive.

[0096] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the control method of the smart pen in the above embodiments.

[0097] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, a radio frequency (RF), and the like, or any suitable combination of the above.

[0098] The above computer readable storage medium can be included in a head-mounted display device, or can exist separately without being assembled into a head-mounted display device.

[0099] The above computer readable storage medium carries one or more programs, which, when executed by the head-mounted display device, cause the head-mounted display device to: detect that the nib contacts a writing plane of a virtual scene of the head-mounted display device, obtain a contact distance between the nib and the writing plane and a moving direction; According to the contact distance and the moving direction, a vibration parameter of the vibration motor is determined; According to the vibration parameter, the vibration motor is controlled to operate, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0100] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0102] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0103] The readable storage medium provided by the application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the control method of the intelligent pen, and can solve the technical problem that the interaction of the virtual scene is poor due to the lack of tactile feedback in the real writing process when writing in the virtual scene. Compared with the prior art, the computer readable storage medium provided by the application has the same beneficial effects as the control method of the intelligent pen provided by the above-mentioned embodiments, and will not be described here.

[0104] The above-mentioned is only part of the embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the application and the content of the specification and drawings are included in the patent protection scope of the application.

Claims

1. A control method of a smart pen, the method comprising: The application is applied to a head-mounted display device connected with a smart pen, a tip of the smart pen is provided with a vibration motor, and a control method of the smart pen comprises the following steps: When it is detected that the tip contacts a writing plane of a virtual scene of the head-mounted display device, the contact distance between the tip and the writing plane and the moving direction are obtained; According to the contact distance and the moving direction, the vibration parameters of the vibration motor are determined; According to the vibration parameters, the vibration motor is controlled to operate, so that the smart pen generates vibration feedback when writing in the virtual scene. 2.The control method of the smart pen of claim 1, wherein, The vibration motor comprises a horizontal vibration motor and a vertical vibration motor, the vibration parameters comprise horizontal vibration parameters and vertical vibration parameters, and the step of determining the vibration parameters of the vibration motor according to the contact distance and the moving direction comprises the following steps: According to the absolute value of the contact distance, the vertical vibration parameters of the vertical vibration motor are calculated, the absolute value is positively correlated with the amplitude of the vertical vibration parameters when the tip contacts the writing plane; The moving speed of the tip based on the moving direction is determined; According to the moving direction and the moving speed, the horizontal vibration parameters of the horizontal vibration motor are determined, the direction of the horizontal vibration parameters is opposite to the moving direction, and the amplitude of the horizontal vibration parameters is negatively correlated with the moving speed. 3.The control method of the smart pen of claim 2, wherein, After the step of determining the horizontal vibration parameters of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameters is opposite to the moving direction, and the amplitude of the horizontal vibration parameters is negatively correlated with the moving speed, the control method of the smart pen comprises the following steps: An inclination angle between the smart pen and the writing plane is obtained; A first horizontal amplitude corresponding to the inclination angle is determined, and a second horizontal amplitude corresponding to the moving speed is determined; The first horizontal amplitude and the second horizontal amplitude are weighted and fused to obtain a target amplitude, and the current amplitude of the horizontal vibration parameters is updated based on the target amplitude. 4.The control method of the smart pen of claim 2, wherein, After the step of determining the horizontal vibration parameters of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameters is opposite to the moving direction, and the amplitude of the horizontal vibration parameters is negatively correlated with the moving speed, the control method of the smart pen further comprises the following steps: A turning point corresponding to a turning time when the moving direction of the tip changes is determined; A third horizontal amplitude corresponding to the turning time is updated based on a preset turning vibration parameter, and the horizontal vibration motor rotates based on the third horizontal amplitude in a time period corresponding to the turning time. 5.The control method of the smart pen of claim 2, wherein, After the step of determining the horizontal vibration parameters of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameters is opposite to the moving direction, and the amplitude of the horizontal vibration parameters is negatively correlated with the moving speed, the control method of the smart pen further comprises the following steps: A horizontal vibration component of the vertical vibration parameters in the horizontal direction is calculated; The horizontal vibration component is superimposed in the horizontal vibration parameters to obtain target horizontal vibration parameters capable of offsetting the horizontal vibration component; Based on the target horizontal vibration parameter and the vertical vibration parameter, the vibration motor is controlled to operate so as to make the smart pen generate vibration feedback when writing in the virtual scene. 6.The control method of the smart pen according to claim 1, wherein, After the step of controlling the vibration motor to operate based on the vibration parameter so as to make the smart pen generate vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises: determining a vibration amplitude in the vibration parameter that is perpendicular to the writing plane; determining a target developing parameter of the to-be-rendered text according to a correspondence between the vibration amplitude and the developing parameter; rendering writing content of the smart pen based on the target developing parameter. 7.The control method of the smart pen of claim 1, wherein, After the step of controlling the vibration motor to operate based on the vibration parameter so as to make the smart pen generate vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises: obtaining an audio playing parameter corresponding to a vibration amplitude of the vibration parameter; outputting a writing sound effect of the smart pen based on the audio playing parameter. 8.The control method of the smart pen according to any one of claims 1 to 7, wherein, The control method of the smart pen further comprises: in response to an erasing instruction, entering an erasing mode and obtaining an erasing distance of the nib when the nib is in contact with the writing plane; determining an erasing vibration parameter of the vibration motor according to the erasing distance; controlling the vibration motor to operate based on the erasing vibration parameter so as to make the smart pen generate erasing vibration feedback.

9. A head-mounted display device, comprising: The head-mounted display device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the control method of the smart pen according to any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the control method of the smart pen according to any one of claims 1 to 8.

Images