Control method of smart pen, head-mounted display device, and storage medium

Abstract

The application discloses a control method of a smart pen, a head-mounted display device and a storage medium, relates to the technical field of electric data processing, and discloses a control method of a smart pen, which is applied to a head-mounted display device, the head-mounted display device is connected with the smart pen, a vibration motor is arranged on a pen point of the smart pen, and the control of the smart pen comprises the following steps: detecting that the pen point is in contact with a writing plane of a virtual scene of the head-mounted display device, acquiring a contact distance between the pen point and the writing plane and a moving direction; determining vibration parameters of the vibration motor according to the contact distance and the moving direction; and controlling the vibration motor to operate according to the vibration parameters, so that the smart pen generates vibration feedback when writing in the virtual scene. Based on this, when a user writes in the virtual scene by using the smart pen, the user can perceive the contact state with the virtual writing plane and the change in movement through the vibration of the pen point, and the tactile experience in the virtual writing process is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric data processing, and particularly relates to a control method of a smart pen, a head-mounted display device and a storage medium. BACKGROUND

[0002] In an XR (Extended Reality) device such as an XR head-mounted display, smart glasses, etc., such as AR (Augmented Reality), VR (Virtual Reality), MR (MixedReality), a user can see virtual objects or scenes that cannot be directly observed by naked eyes.

[0003] The smart pen is a cross-platform general interaction device and is widely used in traditional display terminals such as tablets and computers. The smart pen can also be used as a supplementary interaction tool for an XR scene to help a user more accurately interact with virtual content displayed in the XR device.

[0004] However, when the smart pen is used as a supplementary interaction tool for an XR device to realize air writing in a virtual scene, the smart pen can only draw content, and a user cannot perceive details of a writing action, which reduces the interaction experience of the virtual scene.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent an acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a control method of a smart pen, a head-mounted display device and a storage medium, and aims to solve the technical problem that the interaction of a virtual scene is poor when writing in the virtual scene because of the lack of tactile feedback in a real writing process.

[0007] To achieve the above purpose, the present application provides a control method of a smart pen, applied to a head-mounted display device, the head-mounted display device being connected to the smart pen, a vibration motor being arranged at a pen tip of the smart pen, and the control method of the smart pen comprising the following steps.

[0008] detecting that the pen tip contacts a writing plane of a virtual scene of the head-mounted display device, acquiring a contact distance between the pen tip and the writing plane and a moving direction;

[0009] determining vibration parameters of the vibration motor according to the contact distance and the moving direction;

[0010] controlling the vibration motor to operate according to the vibration parameters, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0011] In an embodiment, the vibration motor comprises a horizontal vibration motor and a vertical vibration motor, the vibration parameter comprises a horizontal vibration parameter and a vertical vibration parameter, and the step of determining the vibration parameter of the vibration motor according to the contact distance and the moving direction comprises:

[0012] calculating the vertical vibration parameter of the vertical vibration motor according to the absolute value of the contact distance, the absolute value being positively correlated with the amplitude of the vertical vibration parameter when the stylus tip and the writing plane are in contact;

[0013] determining a moving speed of the stylus tip when moving based on the moving direction;

[0014] determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed.

[0015] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed, the control method of the smart pen comprises:

[0016] obtaining an inclination angle between the smart pen and the writing plane;

[0017] determining a first horizontal amplitude corresponding to the inclination angle, and determining a second horizontal amplitude corresponding to the moving speed;

[0018] weighting and fusing the first horizontal amplitude and the second horizontal amplitude to obtain a target amplitude, and updating a current amplitude of the horizontal vibration parameter based on the target amplitude.

[0019] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter being opposite to the moving direction, and the amplitude of the horizontal vibration parameter being negatively correlated with the moving speed, the control method of the smart pen further comprises:

[0020] determining a turning point corresponding to a turning time when the moving direction of the stylus tip changes;

[0021] updating a third horizontal amplitude corresponding to the turning time based on a preset turning vibration parameter, and rotating the horizontal vibration motor based on the third horizontal amplitude in a period corresponding to the turning time.

[0022] In an embodiment, after the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter is opposite to the moving direction, and the amplitude of the horizontal vibration parameter is negatively correlated with the moving speed, the control method of the smart pen further comprises:

[0023] calculating a horizontal vibration component of the vertical vibration parameter in a horizontal direction;

[0024] superimposing the horizontal vibration component in the horizontal vibration parameter to obtain a target horizontal vibration parameter capable of offsetting the horizontal vibration component;

[0025] controlling the vibration motor to operate based on the target horizontal vibration parameter and the vertical vibration parameter, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0026] In an embodiment, after the step of controlling the vibration motor to operate based on the vibration parameter, so that the smart pen generates vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises:

[0027] determining a vibration amplitude of the vibration parameter perpendicular to the writing plane;

[0028] determining a target developing parameter of the text to be rendered according to a corresponding relationship between the vibration amplitude and the developing parameter;

[0029] rendering the writing content of the smart pen based on the target developing parameter.

[0030] In an embodiment, after the step of controlling the vibration motor to operate based on the vibration parameter, so that the smart pen generates vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises:

[0031] obtaining an audio playing parameter corresponding to a vibration amplitude of the vibration parameter;

[0032] outputting a writing sound effect of the smart pen when writing based on the audio playing parameter.

[0033] In an embodiment, the control method of the smart pen further comprises:

[0034] in response to an erasing instruction, entering an erasing mode and obtaining an erasing distance of the nib when the nib is in contact with the writing plane;

[0035] determining an erasing vibration parameter of the vibration motor according to the erasing distance;

[0036] controlling the vibration motor to operate based on the erasing vibration parameter, so that the smart pen generates erasing vibration feedback.

[0037] In addition, to achieve the above object, the application further provides a head-mounted display device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the control method of the smart pen.

[0038] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the control method of the smart pen.

[0039] The one or more technical solutions provided by the application have at least the following technical effects:

[0040] Connecting the head-mounted display device with the smart pen, setting a vibration motor on the tip of the smart pen, so that when the tip is detected to contact the writing plane of the virtual scene of the head-mounted display device, the contact distance between the tip and the writing plane and the moving direction are obtained, and then the vibration parameters of the vibration motor are determined according to the obtained information and the vibration motor is controlled to run to generate vibration feedback, so that when the user uses the smart pen to write in the virtual scene, the vibration of the tip is used to enhance the touch experience in the virtual writing process, improve the accurate perception and immersion of the user to the operation of the smart pen, make the writing operation in the virtual scene closer to the real writing experience, and improve the interactive experience when writing in the virtual scene. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0043] Figure 1 The flowchart provided for the first embodiment of the control method of the smart pen of the application;

[0044] Figure 2 The flowchart provided for the second embodiment of the control method of the smart pen of the application;

[0045] Figure 3 The flowchart provided for the fifth embodiment of the control method of the smart pen of the application;

[0046] Figure 4A device structure schematic diagram of a hardware running environment involved in a control method of an intelligent pen in embodiments of the present application.

[0047] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0049] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] In AR, VR, MR and other XR devices such as XR headsets and smart glasses, users can see virtual objects or scenes that cannot be directly observed by the naked eye. As a cross-platform general-purpose interactive device, the intelligent pen is widely used in traditional display terminals such as tablets and computers, and can also be used as a supplementary interactive tool for XR scenes to help users interact more accurately with virtual content displayed in the XR device.

[0051] However, as a supplementary interactive tool for XR devices, the intelligent pen can only draw content when used for air writing in a virtual scene, and users cannot perceive the details of the writing action, which reduces the interactive experience of the virtual scene.

[0052] Therefore, the main solution of the embodiments of the present application is to detect the contact between the pen tip and the writing plane of the virtual scene of the head-mounted display device, obtain the contact distance and movement direction between the pen tip and the writing plane, and determine the vibration parameters of the vibration motor according to the contact distance and the movement direction.

[0053] According to the contact distance and the movement direction, the vibration parameters of the vibration motor are determined.

[0054] According to the vibration parameters, the vibration motor is controlled to operate, so that the intelligent pen generates vibration feedback when writing in the virtual scene.

[0055] Specifically, by setting a vibration motor on the pen tip of the intelligent pen, when the pen tip is detected to contact the writing plane of the virtual scene of the head-mounted display device, the contact distance and movement direction between the pen tip and the writing plane are obtained, and then the vibration parameters of the vibration motor are determined according to the obtained information and the vibration motor is controlled to operate to generate vibration feedback, so that when the user uses the intelligent pen to write in the virtual scene, the vibration of the pen tip enhances the tactile experience in the virtual writing process.

[0056] It should be noted that the execution subject of the embodiment can be a head-mounted display device with data processing, network communication and program running functions, etc. The head-mounted display device includes VR glasses, AR glasses, MR glasses, etc. The following takes the head-mounted display device as an example to describe the embodiment and each of the following embodiments.

[0057] Based on this, the embodiment of the present application provides a control method of a smart pen, which is applied to a head-mounted display device, the head-mounted display device is connected with the smart pen, and a user writes in a virtual scene through the smart pen. Wherein, a nib of the smart pen is provided with a vibration motor, so as to provide a touch experience in the virtual scene through the vibration motor when the user writes.

[0058] Please refer to Figure 1 , Figure 1 which is a flowchart of the first embodiment of the control method of the smart pen of the present application.

[0059] In the embodiment, the control method of the smart pen includes steps S10-S30:

[0060] In step S10, when the nib is detected to contact a writing plane of the virtual scene of the head-mounted display device, the contact distance between the nib and the writing plane and the moving direction are acquired.

[0061] In the embodiment, the head-mounted display device locates and tracks the nib of the smart pen through infrared vision positioning technology, pure vision positioning technology and other similar VR handle position tracking technologies, so as to detect whether the nib of the smart pen falls in the virtual plane that needs to be written in the virtual scene, that is, to determine whether the nib contacts the writing plane. The writing plane is the plane in the virtual scene that needs to be written on the surface of the virtual object. The contact distance is the vertical space distance from the nib to the virtual writing plane, that is, the vertical contact distance in the direction perpendicular to the writing plane; the moving direction is the motion trajectory vector of the nib on the writing plane of the virtual scene, including the direction angle and the motion speed.

[0062] As an optional implementation, after the user wears the head-mounted display device and uses the smart pen, the head-mounted display device captures the infrared signal of the pen tip in real time based on the infrared camera, and converts the captured signal into spatial coordinates. At the same time, the head-mounted display device can determine the three-dimensional coordinates of the writing plane based on the virtual scene constructed by itself. Then, when the user moves the pen tip close to the position of the virtual plane, the coordinates of the pen tip gradually approach the coordinates of the writing plane. When the distance between the pen tip coordinates and the writing plane is less than a preset threshold, such as less than or equal to 0, it is determined that the pen tip and the writing plane are in contact. In the contact state, the head-mounted display device continuously records the change of the pen tip coordinates, and calculates the displacement direction and moving speed in unit time through the difference between adjacent coordinate points. It can be understood that when the user writes, the hand will be lifted after writing a character, and at this time the pen tip is not in contact with the writing plane. Therefore, after detecting that the pen tip and the writing plane are in contact, the head-mounted display device considers that the user is in a writing state, and under the premise that the pen tip is not obviously lifted, such as the distance between the pen tip and the writing plane being within a preset distance, such as less than 0.1 meters, the contact distance and movement direction of the pen tip and the writing plane are continuously collected at this time. The distance between the pen tip coordinates and the coordinates of the writing plane surface can be less than 0, that is, to ensure that the user writes on the virtual writing plane.

[0063] For example, when the user writes a character on the virtual blackboard using the smart pen, the infrared camera of the smart glasses captures the coordinates of the infrared point of the pen tip, and calculates that the vertical distance from the blackboard plane is -0.6 mm. At this time, it is determined that the pen tip is in contact, and the continuous coordinate change is used to determine that the pen tip moves to the right and down, and the speed is 2.5 cm / s.

[0064] It should be noted that the above parameters are only used for explanation and illustration, and are not limited to the present application.

[0065] Step S20, determining the vibration parameter of the vibration motor according to the contact distance and the movement direction.

[0066] In the present embodiment, the vibration parameter is the core parameter for controlling the operation of the vibration motor, including the vibration frequency, the amplitude and the vibration direction, etc. Based on the vibration parameter, the vibration motor feeds back the force parallel to the writing plane to the user through vibration. Alternatively, in addition to the force parallel to the writing plane, the normal writing also includes the force perpendicular to the writing plane, so the vibration motor can also feed back the force parallel to the writing plane and the force perpendicular to the writing plane to the user through vibration. Among them, one vibration motor provided on the pen tip can be used to realize the force in different directions, or multiple vibration motors provided on the pen tip can be used to provide the force in different directions respectively, such as two vibration motors, which respectively provide the force in the parallel direction and the vertical direction based on the operation of the vibration parameter.

[0067] Further, the vibration parameter can be calculated based on the contact distance and the moving direction through a preset mapping relationship, an algorithm model, etc. Specifically, the head-mounted display device calls a pre-stored mapping table, which is established based on a large amount of experimental data and contains the corresponding relationship between the contact distance and the vibration amplitude and the corresponding relationship between the moving direction change rate and the vibration frequency, inputs the obtained contact distance into the mapping table, and thus matches the corresponding vibration amplitude, inputs the calculated moving direction into the mapping table, and thus matches the corresponding vibration frequency. Based on this, the vibration parameter of the vibration motor is obtained.

[0068] Alternatively, the head-mounted display device takes the contact distance and the moving direction change rate as input features based on a dynamic adjustment algorithm of a neural network, and then calculates the vibration amplitude matching the current interaction state in real time through a model trained by historical writing data, such as reducing the distance to reduce the amplitude and changing the direction to increase the vibration frequency, so as to obtain the corresponding vibration parameter.

[0069] It can be understood that when there are multiple vibration motors, the vibration parameter corresponding to each vibration motor needs to be calculated so as to provide forces in different directions when the vibration motor vibrates.

[0070] Step S30: controlling the vibration motor to operate according to the vibration parameter, so that the smart pen generates the vibration feedback when writing in the virtual scene.

[0071] In the embodiment, after the head-mounted display device determines the vibration parameter, the head-mounted display device sends the vibration parameter to the smart pen, so that the vibration motor in the pen tip rotates, thereby generating the vibration feedback when writing in the virtual scene, including the friction force opposite to the moving direction or the friction force opposite to the moving direction and the rebound force perpendicular to the writing plane, so that the user perceives the writing resistance when writing or the writing resistance and the force of the pen tip pressing the paper.

[0072] It should be noted that the head-mounted display device starts to calculate the real-time vibration parameter when the pen tip contacts the writing plane of the virtual scene, so that the user can perceive the physical feedback caused by the vibration of the pen tip of the smart pen in real time when writing in the virtual scene.

[0073] When the pen tip is lifted and leaves the writing plane, the vibration motor of the pen tip stops vibrating. When the pen tip is in a stationary state, the vibration motor also stops vibrating.

[0074] Alternatively, in addition to determining the vibration parameter through the head-mounted display device and controlling the vibration motor to operate based on the vibration parameter to generate the vibration feedback when writing, the smart pen also sets a vibration mode, sets the corresponding vibration mode based on the writing habit of the user, and thus selects the vibration mode to vibrate when the pen tip contacts the writing plane.

[0075] Optionally, simple vibration feedback can also be performed by the pen nib, such as when the active range of the pen nib is small, the pen nib is controlled to vibrate to control the vibration motor to operate.

[0076] The embodiment provides a control method of the smart pen. The pen nib contact distance and the moving direction when the pen nib contacts the writing plane in the virtual scene are detected through the head-mounted display device. The vibration parameter required by the vibration motor is calculated according to the contact distance and the moving direction. Then, the vibration motor at the pen nib is controlled to operate based on the parameter, so that vibration feedback of the writing operation in the virtual scene is realized. In this way, the reality and immersion of virtual writing are improved, and the user can obtain a consistent touch experience as real writing in the virtual scene.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, the vibration motor includes a horizontal vibration motor and a vertical vibration motor, and the vibration parameter includes a horizontal vibration parameter and a vertical vibration parameter. The writing resistance is generated through the horizontal vibration motor, and the pressure of the pen nib pressing the paper is generated through the vertical vibration motor. Based on this, please refer to Figure 2 , step S20 further includes steps S21-S23:

[0078] In step S21, the vertical vibration parameter of the vertical vibration motor is calculated according to the absolute value of the contact distance.

[0079] In the embodiment, when the pen nib contacts the writing plane, the absolute value of the contact distance and the amplitude of the vertical vibration parameter are positively correlated. For example, the absolute value of the contact distance increases by A millimeters, and the vertical vibration amplitude increases by B millimeters, or the contact distance is in the interval a1-a2, and the corresponding vibration amplitude is b1 millimeters, and in the interval a3-a4, the corresponding vibration amplitude is b2 millimeters, where a3>a2, and b2 is greater than b2.

[0080] In other words, when the pen nib penetrates into the writing plane, the vibration intensity in the vertical direction will become larger and larger, that is, the amplitude is proportional to the absolute value of the contact distance. Therefore, the amplitude of the vertical vibration can be calculated through the mapping relationship between the contact distance and the amplitude. For example, the amplitude=k*contact distance, and k is a mapping parameter. In addition, the vibration frequency can also be calculated through the mapping relationship between the contact distance and the vibration frequency. The specific mapping method is not limited in the present application.

[0081] It should be noted that the greater the amplitude, the stronger the impact force of the vibration motor on the shell, and the greater the resistance that the user can feel caused by the vibration.

[0082] In step S22, the moving speed of the pen nib based on the moving direction is determined.

[0083] After obtaining the vertical vibration parameter of the vertical vibration motor, the horizontal vibration parameter including the vibration amplitude and the vibration direction needs to be calculated, so as to generate the writing resistance opposite to the writing direction.

[0084] Therefore, the head-mounted display device needs to determine the moving speed of the pen tip based on the moving direction through the coordinate change speed of the pen tip when moving.

[0085] In step S23, the horizontal vibration parameter of the horizontal vibration motor is determined according to the moving direction and the moving speed.

[0086] In this embodiment, the direction of the horizontal vibration parameter is opposite to the moving direction, and the amplitude of the horizontal vibration parameter is negatively correlated with the moving speed. Therefore, the amplitude corresponding to the moving speed can also be determined by the mapping relationship. In the normal moving process, when the pen tip contacts and moves relative to the writing plane, the vibration intensity is the strongest at the moment of moving, and the vibration intensity decreases after moving and keeps a slight vibration, so as to simulate the paper friction characteristics and generate the writing resistance. When the pen tip moves quickly on the real plane such as paper, the contact time of the pen tip with the surface per unit area is shortened, and the continuous perception intensity of the friction force is reduced. That is, the existence of the friction resistance felt by the finger when quickly drawing on the paper is weaker than the resistance felt when slowly writing. Therefore, in the virtual scene, the moving speed and the amplitude of the horizontal vibration parameter are negatively correlated, so that the physical characteristics in the real writing can be accurately simulated in the virtual scene, the reality of the virtual writing is enhanced, and the interaction experience is improved.

[0087] It should be noted that the horizontal vibration motor generates the writing resistance on the writing plane when vibrating, and therefore the vibration direction can be any direction in the plane. Therefore, to avoid the resistance direction not corresponding to the actual writing direction, the vibration direction of the horizontal vibration motor needs to be determined through the moving direction. For example, when moving to the right, the vibration direction is to the left, so as to generate the writing resistance in the opposite direction. Similarly, in the vertical vibration parameter of the vertical vibration motor, the vibration direction is opposite to the direction of the pen tip of the smart pen.

[0088] The embodiment provides a control method of a smart pen. The horizontal vibration motor and the vertical vibration motor are arranged, vibration control in different directions is realized based on the two different vibration motors, so as to generate the paper resistance in writing and the rebound resistance of the pen tip pressing the paper. Meanwhile, the moving speed and the vibration amplitude in the horizontal direction are inversely proportional, so as to simulate the case that the friction force is smaller when the pen tip quickly slides in the real writing, so that the user can perceive the writing action details in the virtual scene, and the interaction experience in the virtual scene is improved.

[0089] Based on the second embodiment of the present application, in the third embodiment of the present application, in the simulation scene, different inclination angles correspond to different intensities of horizontal vibration, for example, when the pen nib is close to vertical writing, the pressure of the pen is mainly concentrated on the tip contact point of the pen nib, the normal pressure of the pen nib and the paper is larger, the frictional resistance of the paper fiber to the pen nib is more sufficient, and the finger can clearly feel the resistance feedback. When the pen nib is greatly inclined, the pressure of the pen is dispersed to the side contact area of the pen nib, the effective normal pressure is reduced, the frictional resistance of the pen nib and the paper naturally weakens, and the resistance felt by the finger is obviously weaker. Therefore, in this embodiment, when calculating the horizontal vibration parameter of the horizontal vibration motor, the current inclination angle of the pen nib also needs to be analyzed and calculated. Therefore, after step S23, steps S24-S26 are further included:

[0090] Step S24, an inclination angle between the smart pen and the writing plane is obtained.

[0091] Step S25, a first horizontal amplitude corresponding to the inclination angle is determined, and a second horizontal amplitude corresponding to the moving speed is determined.

[0092] Step S26, the first horizontal amplitude and the second horizontal amplitude are weighted and fused to obtain a target amplitude, and the current amplitude of the horizontal vibration parameter is updated based on the target amplitude.

[0093] In this embodiment, the head-mounted display device can determine the inclination angle between the smart pen and the writing plane through image analysis. Then, based on the preset mapping relationship, the first horizontal amplitude corresponding to the inclination angle is calculated. It can be understood that the horizontal amplitude refers to the amplitude of the vibration of the horizontal vibration motor. Similarly, based on the mapping relationship between the moving speed and the amplitude, the second horizontal amplitude is calculated. Finally, the target amplitude corresponding to the current moving speed and the current inclination angle during writing is obtained by weighting and fusing the two amplitudes, and the current amplitude in the horizontal vibration parameter is updated based on the target amplitude, so as to improve the reality of the vibration of the horizontal vibration motor in the virtual scene and improve the interaction experience of the user. The process of calculating based on the mapping relationship is similar to the calculation process of the foregoing embodiments, which will not be described herein. The current amplitude is calculated in real time.

[0094] For example, the amplitude calculated based on the moving speed is C1 millimeters. Under normal circumstances, the horizontal vibration motor vibrates based on the amplitude of C1 millimeters. Since there is a certain inclination between the smart pen and the writing plane, the amplitude calculated based on the inclination angle is C2 millimeters. The target amplitude C=aC1+b*C2 is obtained by weighting and summing the two amplitudes, and finally the horizontal vibration motor vibrates based on the real-time calculated target amplitude C.

[0095] The amplitude of the horizontal vibration is adjusted by the tilt angle of the smart pen, so that the vibration is more consistent with the force scenario in real writing, and the interaction experience in writing in the virtual scenario is improved.

[0096] Based on the second or third embodiment of the present application, in the fourth embodiment of the present application, in addition to the influence of the tilt angle on the real vibration feedback, in the actual writing process, the pen tip slides uniformly in a fixed direction, the friction with the paper surface is stable, the resistance perceived by the hand is smooth and continuous, and when the pen tip turns, the pen tip needs to change the movement direction, at this time, the contact point of the pen tip with the paper surface produces instantaneous extrusion due to the change of direction, the normal pressure becomes larger, and the resistance of the paper fiber to the pen tip is enhanced. In this process, the sliding friction is converted into micro-stuttering friction, and the resistance reaches a peak value in a very short time, and then the pen tip moves towards the new movement direction after the turning, at this time, the friction stabilizes and quickly falls back to a new stable value along the direction.

[0097] Therefore, to improve the vibration feedback effect of the virtual scenario, after step S23, it further includes: determining the time corresponding to the turning point when the moving direction of the pen tip changes, and then updating the third horizontal amplitude corresponding to the turning time based on the preset turning vibration parameters. Wherein, the head-mounted display device pre-stores the preset turning vibration parameters corresponding to the turning point, in other words, each turning point corresponds to a set of vibration parameters, for example, the amplitude first increases and then decreases. Based on the preset turning vibration parameters, the third horizontal amplitude corresponding to the turning time is updated, so that the horizontal vibration motor rotates based on the third horizontal amplitude at the time period corresponding to the turning time, and then rotates based on the amplitude calculated in real time according to the moving speed and the tilt angle.

[0098] By setting corresponding vibration parameters for the turning time of the pen tip, the vibration feedback effect of the user writing in the virtual scenario is further improved, and the interaction experience in writing in the virtual scenario is improved.

[0099] Based on the second, third or fourth embodiment of the present application, in the fifth embodiment of the present application, according to the usage habit of normal people, when the user writes in the virtual scenario, the smart pen generally has a certain tilt angle with the writing plane. At this time, the force generated by the vibration perpendicular to the writing direction includes two parts, one part is the feedback force simulated by the pen tip pressing downward vertically, and the other part is the horizontal force with the writing plane.

[0100] When the smart pen is tilted, the vibration of the vertical vibration motor will be decomposed due to the angle, and an additional horizontal component force will be generated. This horizontal component force is neither generated by the user's active pressing nor required by the simulated sliding resistance. This resistance will make the user perceive a strange lateral force, for example, the user only presses the pen tip downward, but feels that the pen tip is slightly pulled to the left or right. This touch feeling is completely inconsistent with real writing, which breaks the realism of the virtual scenario and reduces the interaction experience.

[0101] Therefore, in the present embodiment, please refer to Figure 3 , after step S23, further comprising steps S40-S60:

[0102] Step S40, calculating the horizontal vibration component of the vertical vibration parameter in the horizontal direction.

[0103] In the present embodiment, the way to calculate the horizontal vibration component of the vertical vibration parameter can be through the decomposition of force, decomposing the force generated by the tilted vertical vibration into the vertical vibration component perpendicular to the writing plane and the horizontal vibration component parallel to the writing plane. For example, the horizontal vibration component is equal to the vertical amplitude multiplied by sinθ when the tilt angle is θ.

[0104] It can be understood that the vibration effect generated by the vertical vibration motor can be considered as the effect in the Z axis, while the vibration effect generated by the horizontal vibration motor is in the XY plane, and when the smart pen is tilted, the vertical vibration motor vibrates to generate forces in the Z axis and the XY plane.

[0105] Step S50, superimposing the horizontal vibration component in the horizontal vibration parameter to obtain the target horizontal vibration parameter capable of canceling the horizontal vibration component.

[0106] In the present embodiment, the force generated by the horizontal vibration motor needs to satisfy: interference force + horizontal motor force = target friction force, where the interference force is the horizontal vibration component, and the horizontal motor force is the force when the horizontal vibration motor vibrates itself. Therefore, the target friction force is the vector sum of the two forces, so the horizontal vibration component is superimposed in the horizontal vibration parameter by adding the forces, so as to cancel this part of the invalid force. For example, the amplitude of the horizontal vibration motor when canceling the interference force is calculated by the amplitude formula of simple harmonic vibration synthesis, and the phase of the horizontal vibration motor when canceling the interference force is calculated by the phase synthesis formula.

[0107] Step S60, controlling the vibration motor to operate based on the target horizontal vibration parameter and the vertical vibration parameter, so as to make the smart pen generate the vibration feedback when writing in the virtual scene.

[0108] After obtaining the target horizontal vibration parameter, the action of step S30 is performed based on the last horizontal vibration parameter and the vertical vibration parameter, that is, controlling the vibration motor to operate, so as to complete the tilt interference while reducing the interference brought by the vertical vibration motor on the premise that the smart pen is tilted with the writing plane.

[0109] Exemplarily, the cancellation process of the horizontal component of the vertical vibration motor is illustrated by data example:

[0110] The current smart pen tilt angle is 30°, and the amplitude A of the vertical vibration motor calculated to the current time is 0.5mm vertical =0.5mm, the interference vibration component in the horizontal direction generated by the vertical vibration motor is:

[0111] A disturb =0.5mm, the interference vibration component in the horizontal direction generated by the vertical vibration motor is: vertical* sinθ, where the phase φ disturb =0, that is, the vertical vibration is along the pen axis, and the horizontal component phase is 0. After substituting the data into the formula, the interference vibration component A disturb in the horizontal direction is 0.25mm.

[0112] In the horizontal vibration parameter, the original horizontal vibration component A horiz =0.4mm is calculated, and the phase is π, which is opposite to the direction of the interference force. Therefore, the amplitude formula can be substituted as follows, so as to obtain the target amplitude A target of the target horizontal vibration parameter:

[0113] ,

[0114] Substituting the above parameters into the formula, A target =0.15mm. Wherein, φ horiz is the original horizontal phase.

[0115] At the same time, the phase calculation formula is applied to obtain the actual phase of the target horizontal vibration parameter.

[0116] ,

[0117] Substituting the above known parameters into the formula, the target phase φ target =π.

[0118] That is, the amplitude of the final target horizontal vibration parameter is 0.15mm, and the phase is π.

[0119] It should be noted that the above examples and the above parameters are only used for explanation and are not limited to the calculation method and the calculation parameters.

[0120] The embodiment calculates the phase difference of the two motors in real time, so as to calculate the vibration amplitude and phase of the horizontal vibration motor when actually vibrating, so as to offset or fuse the horizontal component of the vertical motor, so that the user perceives no interference and the friction force consistent with the real physical effect, solves the problem of force interference when writing in the tilt state, and improves the interaction and immersion of virtual writing.

[0121] Based on any of the above embodiments of the present application, in the sixth embodiment of the present application, after controlling the vibration motor to operate based on the vibration parameters to generate vibration feedback, the vibration force generated by the vibration parameters can also be used to render corresponding pictures or output audio, etc., thereby improving the interactive experience in the virtual scene.

[0122] Specifically, after step S30, the method further includes:

[0123] The vibration amplitude in the vibration parameters perpendicular to the writing plane is determined, then the target developing parameter of the text to be rendered is determined according to the correspondence between the vibration amplitude and the developing parameter, and finally the writing content of the smart pen is rendered based on the display parameter. The developing parameter can be ink bleeding effect, color depth effect, etc. Taking the ink bleeding effect as an example, when the user writes with the smart pen, the ink bleeding effect can be rendered and displayed in real time. The deeper the vertical writing penetrates the writing plane, the thicker the stroke thickness is set, and vice versa. The actual effect of the user's pen is better reflected. The faster the horizontal writing speed, the more likely it is to produce no ink, similar to a dry pen. The interactive experience of the virtual scene is enhanced.

[0124] Further, when the vibration motor of the smart pen vibrates, the audio playback parameter corresponding to the vibration amplitude of the vibration parameter can also be obtained, and then the writing sound effect when the smart pen writes is output based on the audio playback parameter. At the same time of writing, the corresponding sound effect is played synchronously, such as the sound of writing accompanied by the user, which enhances the user's experience. The corresponding audio playback parameter is determined through the correspondence, such as the greater the vibration amplitude, the greater the volume, etc. The correspondence can be a preset relationship, which is not limited in the present application.

[0125] In addition, when the user completes writing and erases the writing content, the vibration parameter can also be determined based on the parameter in the erasing process, so as to control the vibration motor to operate, such as simulating the erasing force by vertical vibration and simulating the erasing speed by horizontal vibration, so that the vibration effect is consistent with that in the writing mode. Therefore, after responding to the erasing instruction, the head-mounted display device enters the erasing mode, then the erasing distance when the pen tip and the writing frequency point are in contact is obtained, and the way of obtaining data is the same as that in the writing process, which is not described herein. Then the erasing vibration parameter of the vibration motor is determined according to the erasing distance, and finally the vibration motor is controlled to operate according to the erasing vibration parameter, so as to generate the effect of the smart pen producing erasing vibration feedback.

[0126] The present embodiment provides corresponding vibration developing effect and vibration sound effect at the same time of writing vibration, thereby improving the interactive experience in the virtual scene, and sets the erasing mode, the vibration logic of which is equivalent to that in the writing mode, thereby further improving the interactive experience in the process of using the smart pen.

[0127] The application provides a head-mounted display device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the control method of the smart pen in the first embodiment.

[0128] Reference will be made to the following description Figure 4 which shows a structural schematic diagram of a head-mounted display device suitable for implementing the embodiments of the application. Figure 4 The head-mounted display device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0129] As shown in Figure 4 , the head-mounted display device can include a processing apparatus 1001 (for example, a central processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the head-mounted display device are also stored. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the head-mounted display device to perform wireless or wired communication with other devices to exchange data. Although the head-mounted display device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0130] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0131] The head-mounted display device provided by the present application adopts the control method of the smart pen in the above-mentioned embodiments, and can solve the technical problem that the interaction of the virtual scene is poor due to the lack of tactile feedback in the real writing process when writing in the virtual scene. Compared with the prior art, the head-mounted display device provided by the present application has the same beneficial effects as the control method of the smart pen provided by the above-mentioned embodiments, and other technical features in the head-mounted display device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0132] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0134] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the control method of the smart pen in the above-mentioned embodiments.

[0135] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, a radio frequency (RF), etc., or any suitable combination of the above.

[0136] The above computer readable storage medium may be contained in the head-mounted display device, or may exist separately without being assembled into the head-mounted display device.

[0137] The above computer readable storage medium carries one or more programs, which, when executed by the head-mounted display device, cause the head-mounted display device to: detect that the pen tip is in contact with a writing plane of a virtual scene of the head-mounted display device, obtain a contact distance between the pen tip and the writing plane and a moving direction;

[0138] According to the contact distance and the moving direction, determine a vibration parameter of the vibration motor;

[0139] According to the vibration parameter, control the vibration motor to operate, so that the smart pen generates vibration feedback when writing in the virtual scene.

[0140] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0141] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0142] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0143] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the control method of the smart pen, and can solve the technical problem that the interaction of the virtual scene is poor due to the lack of tactile feedback in the real writing process when writing in the virtual scene. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the control method of the smart pen provided by the above-mentioned embodiments, and will not be described here.

[0144] The above merely provides part of embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and contents of the present application specification and drawings are included in the patent protection scope of the present application.

Claims

1. A control method of a smart pen, the method comprising: The application is applied to a head-mounted display device connected to a smart pen, a tip of the smart pen is provided with a vibration motor, the vibration motor includes a horizontal vibration motor and a vertical vibration motor, and a control method of the smart pen includes: When it is detected that the tip contacts a writing plane of a virtual scene of the head-mounted display device, a contact distance between the tip and the writing plane and a moving direction are acquired; According to the contact distance and the moving direction, vibration parameters of the vibration motor are determined, including: According to an absolute value of the contact distance, a vertical vibration parameter of the vertical vibration motor is calculated, when the tip contacts the writing plane, the absolute value is positively correlated with an amplitude of the vertical vibration parameter; A moving speed of the tip when moving based on the moving direction is determined; According to the moving direction and the moving speed, a horizontal vibration parameter of the horizontal vibration motor is determined, a direction of the horizontal vibration parameter is opposite to the moving direction, and an amplitude of the horizontal vibration parameter is negatively correlated with the moving speed; An inclination angle between the smart pen and the writing plane is acquired; A first horizontal amplitude corresponding to the inclination angle is determined, and a second horizontal amplitude corresponding to the moving speed is determined; A target amplitude is obtained by weighted fusion of the first horizontal amplitude and the second horizontal amplitude, and a current amplitude of the horizontal vibration parameter is updated based on the target amplitude; According to the vibration parameters, the vibration motor is controlled to operate, so that the smart pen generates vibration feedback when writing in the virtual scene, and the vibration parameters include the vertical vibration parameter and the horizontal vibration parameter. 2.The control method of the smart pen of claim 1, wherein, After the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter is opposite to the moving direction, and the amplitude of the horizontal vibration parameter is negatively correlated with the moving speed, the control method of the smart pen further includes: A turning point corresponding to a turning time when the moving direction of the tip changes is determined; A third horizontal amplitude corresponding to the turning time is updated based on a preset turning vibration parameter, and the horizontal vibration motor rotates based on the third horizontal amplitude in a period corresponding to the turning time. 3.The control method of the smart pen of claim 1, wherein, After the step of determining the horizontal vibration parameter of the horizontal vibration motor according to the moving direction and the moving speed, the direction of the horizontal vibration parameter is opposite to the moving direction, and the amplitude of the horizontal vibration parameter is negatively correlated with the moving speed, the control method of the smart pen further includes: A horizontal vibration component of the vertical vibration parameter in a horizontal direction is calculated; The horizontal vibration component is superimposed in the horizontal vibration parameter to obtain a target horizontal vibration parameter capable of offsetting the horizontal vibration component; Based on the target horizontal vibration parameter and the vertical vibration parameter, the vibration motor is controlled to operate, so that the smart pen generates vibration feedback when writing in the virtual scene. 4.The control method of the smart pen of claim 1, wherein, After the step of controlling the vibration motor to operate according to the vibration parameters, so that the smart pen generates vibration feedback when writing in the virtual scene, the control method of the smart pen further includes: determining a vibration amplitude value in the vibration parameter perpendicular to the writing plane; determining a target developing parameter of the to-be-rendered text according to a correspondence between the vibration amplitude value and the developing parameter; rendering the writing content of the smart pen based on the target developing parameter. 5.The control method of the smart pen of claim 1, wherein, After the step of controlling the vibration motor to operate according to the vibration parameter to make the smart pen generate vibration feedback when writing in the virtual scene, the control method of the smart pen further comprises: obtaining an audio playing parameter corresponding to the vibration amplitude value of the vibration parameter; outputting a writing sound effect of the smart pen when writing based on the audio playing parameter. 6.The control method of a smart pen according to any one of claims 1 to 5, wherein, The control method of the smart pen further comprises: in response to an erasing instruction, entering an erasing mode and obtaining an erasing distance of the nib when the nib is in contact with the writing plane; determining an erasing vibration parameter of the vibration motor according to the erasing distance; controlling the vibration motor to operate according to the erasing vibration parameter to make the smart pen generate erasing vibration feedback.

7. A head-mounted display device, comprising: The head-mounted display device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the control method of the smart pen according to any one of claims 1 to 6.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the control method of the smart pen according to any one of claims 1 to 6.

Images