Writing method, intelligent pen, intelligent glasses and computer readable storage medium

Abstract

The invention discloses a writing method, an intelligent pen, intelligent glasses and a computer readable storage medium, and belongs to the technical field of human-computer interaction. According to the invention, a communication connection link is established with the intelligent glasses; based on the communication connection link, using state information, sent by the intelligent glasses, of the intelligent pen is dynamically received, and the using state information is generated by tracking the intelligent pen through a camera of the intelligent glasses; according to the use state information, the vibration characteristics of the pen point part of the intelligent pen are dynamically adjusted, so that the vibration characteristics are matched with the use state information, real-time writing feedback is truly and accurately provided for the user, and the writing efficiency of the user in the virtual space is improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a writing method, a smart pen, smart glasses and a computer-readable storage medium. Background Art

[0002] With the development of technologies such as virtual reality and augmented reality, users can obtain more and more complete information in virtual space. While obtaining information, they also need to input information. Compared with keyboard input, it is more efficient and convenient for users to input information using smart pens, which can meet the needs of a large number of text and painting information interaction scenarios. Users can use smart pens to write freely in virtual space.

[0003] Generally speaking, the virtual system needs the tip of the smart pen to touch the virtual interactive interface before it can recognize the information input by the user. However, when the user writes in the virtual space, the smart pen is used in the air, and the user cannot feel the real touch of the pen touching the real object. Therefore, the user does not know whether the tip of the smart pen touches the virtual interactive interface. This will bring about the problem of invalid writing. That is, the user waves the smart pen to write, but finds that the content has not been successfully entered in the virtual interactive interface. Therefore, the user needs to repeatedly adjust and re-enter the content, which seriously reduces the user's writing efficiency in the virtual space. Summary of the invention

[0004] The main purpose of this application is to provide a writing method, a smart pen, smart glasses and a computer-readable storage medium, aiming to solve the technical problem that it is impossible to provide users with real-time writing feedback truthfully and accurately, resulting in low writing efficiency of users in virtual space.

[0005] To achieve the above objectives, the present application provides a writing method, which is applied to a smart pen, comprising:

[0006] Establishing a communication link with the smart glasses;

[0007] Based on the communication connection link, dynamically receive usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through a camera of the smart glasses;

[0008] The vibration characteristics of the pen tip of the smart pen are dynamically adjusted according to the usage status information so that the vibration characteristics match the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

[0009] Optionally, the use state information includes one of idle state information, writing state information, pen down state information and pen up state information, and the step of dynamically adjusting the vibration characteristics of the pen tip part of the smart pen according to the use state information includes:

[0010] If the usage state information is idle state information, setting the vibration characteristic to a first preset vibration characteristic;

[0011] If the usage status information is writing status information, the vibration feature is set to a second preset vibration feature, wherein the vibration frequency of the second preset vibration feature is greater than the vibration frequency of the first preset vibration feature, and / or the vibration amplitude of the second preset vibration feature is greater than the vibration amplitude of the first preset vibration feature;

[0012] If the use status information is the writing status information, gradually increasing the vibration characteristic from the first preset vibration characteristic to the second preset vibration characteristic within a preset time period;

[0013] If the usage status information is pen-lifting status information, the vibration characteristic is gradually reduced from the second preset vibration characteristic to the first preset vibration characteristic within a preset time period.

[0014] Optionally, before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:

[0015] Dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen;

[0016] The second posture information is sent to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

[0017] Optionally, before the step of establishing a communication connection link with the smart glasses, the method further includes:

[0018] Acquire pressure parameters of the smart pen housing;

[0019] If the pressure parameter is greater than a preset pressure parameter threshold, the step of establishing a communication connection link with the smart glasses is performed.

[0020] In addition, to achieve the above-mentioned purpose, the present application also provides a writing method, which is applied to smart glasses, comprising:

[0021] After establishing a communication connection link with the smart glasses, tracking the features on the housing of the smart pen through the camera of the smart glasses to determine the first position information of the smart pen;

[0022] constructing a three-dimensional virtual image of the smart pen according to the first posture information;

[0023] generating usage status information of the smart pen according to the three-dimensional virtual image;

[0024] The usage status information is sent to the smart pen, so that the smart pen dynamically adjusts the vibration characteristics of the pen tip part of the smart pen according to the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

[0025] Optionally, the step of constructing the three-dimensional virtual image of the smart pen according to the first posture information includes:

[0026] Receiving second position information sent by the smart pen, wherein the first position information and the second position information are the position of the smart pen measured at the same time node;

[0027] The first posture information of the first preset weight and the second posture information of the second preset weight are summed to obtain third posture information, and a three-dimensional virtual image of the smart pen is constructed according to the third posture information.

[0028] Optionally, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image comprises:

[0029] Determine the distance between the pen tip of the three-dimensional virtual image and the virtual image interaction interface, and the change trend of the distance, wherein the change trend includes an increasing trend and a decreasing trend;

[0030] If the distance matches the first preset distance, generating writing status information;

[0031] If the distance matches the second preset distance, and the change trend is an increasing trend, then generating pen lifting state information;

[0032] If the distance matches the second preset distance, and the change trend is a decreasing trend, then generating pen-down state information;

[0033] If the distance matches a third preset distance, idle state information is generated, wherein the second preset distance is greater than the first preset distance, and the third preset distance is greater than the second preset distance.

[0034] The present application also provides a smart pen, which is a physical device. The smart pen includes: a memory, a processor, and a program of the writing method stored in the memory and executable on the processor. When the program of the writing method is executed by the processor, the steps of the writing method described above can be implemented.

[0035] The present application also provides a pair of smart glasses, which is a physical device. The smart glasses include: a memory, a processor, and a program of the writing method stored in the memory and executable on the processor. When the program of the writing method is executed by the processor, the steps of the writing method described above can be implemented.

[0036] The present application provides a writing method, a smart pen, smart glasses and a computer-readable storage medium. By receiving the usage status information of the smart pen generated by the smart glasses, the vibration characteristics of the pen tip of the smart pen (the vibration characteristics include vibration frequency and vibration amplitude) are adjusted in real time according to the usage status information, so as to simulate the reaction force given to the pen by the physical medium when the user writes on the physical medium, thereby giving the user tactile feedback. When the pen in the virtual world does not come into contact with the virtual interactive interface, the user can truly feel it and promptly drop the pen tip of the virtual world pen accurately onto the virtual interactive interface, instead of realizing the need to adjust the smart pen until the written content does not appear on the virtual interactive interface after writing for a period of time, thereby providing the user with real-time writing feedback truly and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A system block diagram of the smart glasses involved in the writing method of this application;

[0040] Figure 2 A system block diagram of the smart pen involved in the writing method of this application;

[0041] Figure 3 A schematic diagram of the flow chart of the first embodiment of the writing method of the present application;

[0042] Figure 4 This is a detailed flowchart of step S300 in the second embodiment of the writing method of this application;

[0043] Figure 5 A schematic diagram of a third embodiment of the writing method of the present application;

[0044] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the smart glasses in this embodiment.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] With the development of technologies such as virtual reality and augmented reality, users can obtain more and more complete information in virtual space. While obtaining information, they also need to input information. Compared with keyboard input, it is more efficient and convenient for users to input information using smart pens, which can meet the needs of a large number of text and painting information interaction scenarios. Users can use smart pens to write freely in virtual space.

[0048] Generally speaking, the virtual system needs the tip of the smart pen to touch the virtual interactive interface before it can recognize the information input by the user. However, when the user writes in the virtual space, the smart pen is used in the air, and the user cannot feel the real touch of the pen touching the real object. Therefore, the user does not know whether the tip of the smart pen touches the virtual interactive interface. This will bring about the problem of invalid writing. That is, the user waves the smart pen to write, but finds that the content has not been successfully entered in the virtual interactive interface. Therefore, the user needs to repeatedly adjust and re-enter the content, thereby reducing the user's writing efficiency in the virtual space.

[0049] The present application provides a writing method, a smart pen, smart glasses and a computer-readable storage medium. By receiving the usage status information of the smart pen generated by the smart glasses, the vibration characteristics of the pen tip of the smart pen (the vibration characteristics include vibration frequency and vibration amplitude) are adjusted in real time according to the usage status information, so as to simulate the reaction force given to the pen by the physical medium when the user writes on the physical medium, thereby giving the user tactile feedback. When the pen in the virtual world does not come into contact with the virtual interactive interface, the user can truly feel it and promptly drop the pen tip of the virtual world pen accurately onto the virtual interactive interface, instead of realizing the need to adjust the smart pen until the written content does not appear on the virtual interactive interface after writing for a period of time, thereby providing the user with real-time writing feedback truly and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space.

[0050] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0051] Reference Figure 1 , Figure 1This is a block diagram of the smart glasses system structure involved in the embodiment of the present application. The smart glasses may include an application processor, such as a central processing unit (CPU), a wireless communication module, a graphics renderer, a 6DoF (Degrees of Freedom) camera module and a virtual image distance adjustment module connected to the application processor through a communication bus. In addition, the smart glasses may also include a memory, which may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. Among them, the wireless communication module is used to interact with the smart pen for data, the graphics renderer is used to project the virtual image into the display module of the smart glasses, the virtual image distance adjustment module is used to adjust the display virtual image distance, and the 6DoF camera module is used to track the smart pen to obtain the posture information of the smart pen. Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the smart glasses, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The above-mentioned smart glasses are also not limited to smart products such as AR (Augmented Reality) glasses, XR (Extended Reality) glasses, MR (Mixed Reality) glasses or other head-mounted display devices.

[0052] Reference Figure 2 , Figure 2This is a block diagram of the smart pen system structure involved in the embodiment of the present application. The smart pen may include a microprocessor, such as a central processing unit (CPU), a wireless communication module, a tactile feedback module and an IMU (Inertial Measurement Unit) module connected to the microprocessor through a communication bus. In addition, it may also include an IR LED (Infrared Light Emitting Diode) array module and a memory. The memory may be a high-speed random access memory or a stable non-volatile memory, such as a disk storage. Among them, the wireless communication module is used to interact with the smart glasses for data, the tactile feedback module is used to adjust the vibration characteristics of the smart pen (the vibration characteristics include vibration frequency and vibration amplitude) to give the user tactile feedback, and the IMU module is used to collect the position and posture of the smart pen itself in real time. The IR LED array module may include a plurality of IR LED lights and drivers that are three-dimensionally distributed on the smart pen, so that the 6DOF camera module of the smart glasses can recognize the smart pen when the user holds the pen in different postures or at different angles. The IR LED array module may also be replaced with a highly reflective pattern or feature object, so that the smart glasses can track the smart pen. Even in a dark environment, the 6DoF camera of the smart glasses can clearly recognize the smart pen. Those skilled in the art will understand that Figure 2 The structure shown in the figure does not constitute a limitation to the smart pen, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] Embodiment 1

[0054] Reference Figure 3 The present application provides a writing method. In a first embodiment of the writing method, in this embodiment, the writing method is applied to a smart pen, and the method comprises the following steps:

[0055] Step S100, establishing a communication connection link with the smart glasses;

[0056] In this embodiment, the smart pen needs to exchange data with the smart glasses. Before the data exchange, the smart pen needs to establish a communication connection link with the smart glasses, and the communication connection link can be a wireless connection. It should be noted that the writing panel used by the smart pen in the writing space can be a virtual writing panel, and of course it can also be a real writing panel, that is, after the smart glasses are connected with the smart pen, a writing panel for writing with the smart pen is established, and the writing action of the smart pen on the writing panel is recognized.

[0057] Step S200, dynamically receiving usage status information of the smart pen sent by the smart glasses based on the communication connection link, wherein the usage status information is generated by tracking the smart pen through a camera of the smart glasses;

[0058] In this embodiment, the smart glasses can use a graphics renderer to project a virtual image on the display module, and build a virtual space with the help of the unity engine. The smart glasses will generate a virtual interactive interface in the virtual space, and at the same time, the smart pen will be reconstructed in three dimensions in the virtual space to generate a three-dimensional virtual image. That is to say, the motion posture of the smart pen in the real world and the motion posture of the three-dimensional virtual image of the smart pen in the virtual space are synchronized. When the user is ready to input content by handwriting, the virtual distance adjustment module will adjust the display virtual image from far to near. For example, when the display virtual image is far away, it mainly provides the user with a display function. Therefore, when the display virtual image is far away, the distance from the smart glasses can be 2 meters, and when the display virtual image is near, a virtual interactive interface will be generated to facilitate the user to write in the virtual space. At this time, the distance between the virtual interactive interface and the smart glasses can be 0.5 meters. The above distance can also be set to other values ​​to better match the most comfortable writing distance for the user. This embodiment does not make specific limitations.

[0059] In this embodiment, the smart glasses track the smart pen through a camera, thereby dynamically obtaining the position information of the smart pen, and generating a three-dimensional virtual image of the smart pen in the virtual space according to the position information. The use status information of the smart pen can be determined through the three-dimensional virtual image. The smart pen dynamically receives the use status information sent by the smart glasses through a communication connection link. The use status information can be one of writing status information and idle status information. When the tip of the three-dimensional virtual image touches the virtual interactive interface, the smart glasses can generate writing status information. If the use status information received by the smart pen is writing status information, it means that the user's writing content in the virtual space can be displayed on the virtual interactive interface in a timely manner. When the tip of the three-dimensional virtual image does not touch the virtual interactive interface and there is a certain distance from the virtual interactive interface, the smart glasses can generate idle status information. If the use status information received by the smart pen is idle status information, it means that the user's writing content in the virtual space cannot be displayed on the virtual interactive interface.

[0060] Optionally, the usage status information can also be pen-down status information or pen-lifting status information. For example, the process of the pen tip of the three-dimensional virtual image moving from a distance relatively far from the virtual interaction interface to contacting the virtual interaction interface is the pen-down process of the three-dimensional virtual image, and the smart glasses can generate pen-down status information. Conversely, the process of the pen tip of the three-dimensional virtual image moving from contacting the virtual interaction interface to a distance relatively far from the virtual interaction interface is the pen-lifting process of the three-dimensional virtual image, and the smart glasses can generate pen-lifting status information.

[0061] After step S200, step S300 is executed to dynamically adjust the vibration characteristics of the pen tip of the smart pen according to the usage status information so that the vibration characteristics match the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

[0062] Exemplarily, the tactile feedback module of the smart pen may include a piezoelectric feedback device or a vibration motor. The piezoelectric feedback device may be a piezoelectric ceramic. The piezoelectric ceramic has an inverse piezoelectric effect. When an electric field is applied to the piezoelectric sheet, the piezoelectric ceramic sheet will produce mechanical deformation, and its strain is proportional to the electric field strength. The stronger the applied electric field strength, the greater the vibration amplitude. The vibration characteristics of the tip of the smart pen can be adjusted by the piezoelectric ceramic or the vibration motor, and the sound emitted by the piezoelectric ceramic when vibrating is extremely clear. Due to its ultra-thin thickness, light weight and no contacts, there is no noise. At the same time, the operating current is small and it has low power consumption characteristics.

[0063] In this embodiment, if the usage status information received by the smart pen is writing status information, it indicates that the user is writing on the virtual interactive interface, or is preparing to write. In order to simulate the tactile feeling of the user when writing on the physical medium in reality, the pen tip of the smart pen can start to vibrate to simulate the reaction force of the physical medium on the smart pen in reality. If the smart pen is already in a vibrating state, the original vibration characteristics can be maintained after receiving the writing status information. If the usage status information received by the smart pen is idle status information, it indicates that the user stops writing on the virtual interactive interface, and the pen does not touch the virtual interactive interface in the virtual space. Therefore, in order to simulate the tactile feeling of the user in this scenario, the pen tip of the smart pen can stop vibrating, or the vibration characteristics (including vibration frequency and vibration amplitude) can be set to a value that the user cannot feel the vibration of the smart pen.

[0064] Optionally, if the usage status information received by the smart pen is the pen-lifting status information, indicating that the user is lifting the smart pen in the virtual space, in order to simulate the process of reducing the reaction force of the physical medium on the smart pen when the user lifts the pen from the physical medium in reality, the vibration frequency and / or vibration amplitude of the pen tip of the smart pen can be gradually reduced. On the contrary, if the usage status information received by the smart pen is the pen-dropping status information, indicating that the user is dropping the smart pen in the virtual space, in order to simulate the process of increasing the reaction force of the physical medium on the smart pen when the user drops the pen on the physical medium in reality, the vibration frequency and / or vibration amplitude of the pen tip of the smart pen can be slowly increased.

[0065] For further information, please refer to Figure 4The use status information includes one of idle state information, writing state information, pen-down state information and pen-up state information. The step S300 of dynamically adjusting the vibration characteristics of the pen tip of the smart pen according to the use status information includes:

[0066] Step S310, if the usage state information is idle state information, setting the vibration characteristic to a first preset vibration characteristic;

[0067] Step S320: If the usage status information is writing status information, the vibration feature is set to a second preset vibration feature, wherein the vibration frequency of the second preset vibration feature is greater than the vibration frequency of the first preset vibration feature, and / or the vibration amplitude of the second preset vibration feature is greater than the vibration amplitude of the first preset vibration feature;

[0068] Step S330: if the use status information is the writing status information, gradually increasing the vibration characteristic from the first preset vibration characteristic to the second preset vibration characteristic within a preset time period;

[0069] Step S340: if the use status information is pen-lifting status information, the vibration characteristic is gradually reduced from the second preset vibration characteristic to the first preset vibration characteristic within a preset time period.

[0070] In this embodiment, the first preset vibration feature is the vibration feature of the pen tip of the smart pen when the three-dimensional virtual image of the smart pen does not touch the virtual interactive interface. For example, the vibration frequency can be zero or other values ​​less than the preset vibration frequency threshold. And / or, the vibration amplitude can be zero or other values ​​less than the preset vibration amplitude threshold, which is not limited in this embodiment. The second preset vibration feature is the vibration feature of the pen tip of the smart pen when the three-dimensional virtual image of the smart pen touches the virtual interactive interface. The vibration frequency of the second preset vibration feature is greater than the vibration frequency of the first preset vibration feature, and / or the vibration amplitude of the second preset vibration feature is greater than the vibration amplitude of the first preset vibration feature. The technicians in this technical field can set it by themselves to truly simulate the real touch when the pen touches the physical medium in reality. The preset duration can be 0.1 seconds or other values, which can be set by the technicians in this technical field. The idle state information is matched with the first preset vibration feature, and the writing state information is matched with the second preset vibration feature. When the use state information is the writing state information, in order to make the vibration feature match the writing state information, the vibration feature of the pen tip of the smart pen is gradually increased from the first preset vibration feature to the second preset vibration feature, so that the vibration feature of the pen tip of the smart pen is smoothly increased (the vibration frequency and / or vibration amplitude of the pen tip of the smart pen is slowly increased), rather than being directly set from the first preset vibration feature to the second preset vibration feature. When the use state information is the pen lifting state information, in order to make the vibration feature match the pen lifting state information, the vibration feature of the pen tip of the smart pen is gradually reduced from the second preset vibration feature to the first preset vibration feature (the vibration frequency and / or vibration amplitude of the pen tip of the smart pen is slowly reduced), so that the vibration feature of the pen tip of the smart pen is smoothly reduced, rather than being directly set from the second preset vibration feature to the first preset vibration feature. In this embodiment, the smart pen adjusts the vibration characteristics of the pen tip in real time according to the usage status information sent by the smart glasses to provide tactile feedback to the user. At the same time, when the three-dimensional virtual image makes a pen-lifting or pen-putting action, the smart pen smoothly adjusts the vibration characteristics of the pen tip to reduce the sense of frustration caused by the sudden change of the vibration characteristics, thereby simulating the tactile feeling of the user when lifting and putting the pen down, and further ensuring that real-time writing feedback is provided to the user in a real and accurate manner.

[0071] In a possible implementation manner, before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:

[0072] Step A10, dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen;

[0073] Those skilled in the art will know that the inertial sensor is an IMU (Inertial Measurement Unit) sensor.

[0074] Step A20, sending the second posture information to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information, and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

[0075] In this embodiment, an IMU module is installed on the smart pen. Through the inertial sensor of the IMU module, the second posture information of the smart pen itself can be dynamically detected, and the second posture information is sent to the smart glasses through the communication connection link. The smart glasses track the smart pen through the camera to obtain the first posture information of the smart pen, and then build a three-dimensional virtual image of the smart pen in combination with the second posture information sent by the smart pen. In this embodiment, the smart pen sends its own posture information to the smart glasses through the communication connection link, which provides a data basis for the smart glasses to build a three-dimensional virtual image of the smart pen.

[0076] In an practicable manner, before the step of establishing a communication connection link with the smart glasses, the method further includes:

[0077] Step B10, obtaining the pressure parameter of the smart pen housing;

[0078] Step B20: If the pressure parameter is greater than the preset pressure parameter threshold, executing the step of establishing a communication connection link with the smart glasses.

[0079] In this embodiment, it is necessary to determine whether the user is using the smart pen. After determining that the user is using the smart pen, the smart pen and the smart glasses establish a communication connection link. At this time, a pressure sensor can be added to the housing of the smart pen to detect the change of the pressure parameter at the housing of the smart pen in real time. The preset pressure parameter threshold is set by the technicians in this technical field to accurately determine whether the smart pen is being used. When the smart pen is picked up (held) by the user, the pressure parameter of the smart pen housing is greater than the preset pressure parameter threshold, then it can be determined that the user is using the smart pen, and the smart pen and the smart glasses establish a communication connection link. When the smart pen is put down (not held) by the user, the pressure value of the smart pen housing is less than or equal to the preset pressure parameter threshold, then it is determined that the smart pen is not in use, and the smart pen is disconnected from the smart glasses. Optionally, a button or switch can be installed on the smart pen. When the user needs to use the smart pen, the smart pen can be manually turned on to establish a communication connection link between the smart pen and the smart glasses. This embodiment automatically controls the communication connection state between the smart pen and the smart glasses, as well as the opening and closing of the smart pen, by detecting the pressure parameters of the smart pen housing, thereby reducing the power consumption of the smart pen while improving the convenience of writing for users in the virtual space.

[0080] In this embodiment, the user uses a smart pen in a virtual space, and the smart glasses generate usage status information of the smart pen according to the three-dimensional virtual image of the smart pen in the virtual space, and share it with the smart pen in real time. The smart pen dynamically adjusts the vibration characteristics of the pen tip according to the corresponding usage status information, so that the user can feel the touch of the pen when it contacts the physical medium when writing in the virtual world. The user can accurately judge whether he has performed effective writing based on the change in touch, and promptly and accurately place the pen tip of the virtual world on the virtual interactive interface. Therefore, this embodiment realizes the real-time and accurate provision of real-time writing feedback to the user, thereby improving the user's writing efficiency in the virtual space.

[0081] Embodiment 2

[0082] Please refer to Figure 5 The present application also provides a writing method of the second embodiment. In this embodiment, the writing method is applied to smart glasses, and the method includes the following steps:

[0083] Step S400, after establishing a communication connection link with the smart glasses, tracking the feature on the housing of the smart pen through the camera of the smart glasses to determine the first position information of the smart pen;

[0084] Step S500, constructing a three-dimensional virtual image of the smart pen according to the first posture information;

[0085] Step S600, generating usage status information of the smart pen according to the three-dimensional virtual image;

[0086] Step S700: sending the usage status information to the smart pen, so that the smart pen adjusts the vibration characteristics of the pen tip of the smart pen according to the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

[0087] In this embodiment, the feature object may be an IR LED array module on the housing of the smart pen. The IR LED array module may include a plurality of IR LED lights and drivers that are three-dimensionally distributed on the smart pen, so that the 6DOF camera module of the smart glasses can recognize the feature object when the user holds the pen in different postures or at different angles. In addition, a highly reflective pattern may be used as a substitute, so that the smart glasses can track the smart pen, and the 6DoF camera of the smart glasses can clearly recognize the smart pen even in a dark environment. The first position information includes the real-time position of the smart pen measured by the smart glasses. The smart glasses track the smart pen through the 6DoF camera module, take photos or videos of the smart pen, and then calculate the first position information of the smart pen through the SLAM (Simultaneous Localization and Mapping) algorithm. The smart glasses may first calibrate the time and position of the 6DoF camera module and the display module in their own display coordinate system. Initially, the position of the three-dimensional virtual image of the smart pen in the display coordinate system may be the same as that of the display module. The smart glasses obtain the first position information of the smart pen, obtain the change of the position and posture of the smart pen according to the information, and reconstruct the three-dimensional virtual image of the smart pen. When a user wears smart glasses or other head-mounted display devices, he may not be able to see the position of the pen, which makes it difficult for the user to write. In one scenario, the user is writing content on a virtual interactive interface. For some reason, the user pauses writing. When the user wants to continue writing, he may not be able to find the position where he paused writing last time. The three-dimensional virtual image of the smart pen is displayed in real time in the virtual space, and the position of the three-dimensional virtual image can be prompted.

[0088] When the smart pen moves and changes its posture in the real world, the corresponding three-dimensional virtual image will also move in the display coordinate system and undergo the same posture change. At this time, the usage status information of the smart pen can be determined by calculating the displacement of the three-dimensional virtual image pen tip. Preferably, the usage status information of the smart pen can be determined by calculating the distance between the three-dimensional virtual image pen tip and the virtual interactive interface, and then the usage status information is sent to the smart pen. The smart pen adjusts the vibration characteristics of the smart pen tip in real time according to the usage status information.

[0089] This embodiment obtains the position information of the smart pen in real time, reconstructs the smart pen in the virtual space, and enables the user to accurately determine the writing position through the three-dimensional virtual image of the smart pen, thereby improving the convenience of the user's writing in the virtual world. In addition, by sending the usage status information of the smart pen to the smart pen, the smart pen adjusts the vibration characteristics of the pen tip part (the vibration characteristics include vibration frequency and vibration amplitude) in real time to match the vibration characteristics with the usage status information, so that the user can accurately judge whether he has performed effective writing in the virtual space, thereby realizing the real-time writing feedback provided to the user in a real and accurate manner, and improving the user's writing efficiency in the virtual space.

[0090] In a possible implementation, the step of constructing the three-dimensional virtual image of the smart pen according to the first posture information includes:

[0091] Step C10, receiving second position information sent by the smart pen, wherein the first position information and the second position information are the position of the smart pen measured at the same time node;

[0092] Step C20, calculating the sum of the first posture information of the first preset weight and the second posture information of the second preset weight to obtain third posture information, and constructing a three-dimensional virtual image of the smart pen according to the third posture information.

[0093] In this embodiment, the second posture information is the posture information of the smart pen collected in real time by the IMU module, and is sent to the smart glasses after collection. The first posture information and the second posture information respectively include the posture of the smart pen measured by the smart glasses and the smart pen at the same time node. Since the first posture information and the second posture information may have certain errors, the two groups of posture information can be combined for comprehensive judgment at this time. Preset weights can be set for the two groups of postures respectively, and the first posture information of the first preset weight and the second posture information of the second preset weight are summed to calculate the third posture information, and then the third posture information is used to construct the three-dimensional virtual image of the smart pen. In this embodiment, the first preset weight and the second preset weight can be the same or different, and the sum of the first preset weight and the second preset weight can be one. This embodiment receives the second posture information measured by the smart pen itself, and determines the posture of the smart pen by combining the first posture information and the second posture information, thereby reducing the error and improving the accuracy of the construction of the three-dimensional virtual image of the smart pen.

[0094] In another possible implementation, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:

[0095] Step D10, simultaneously tracking the features on the housing of the smart pen by using a preset number of cameras on the smart glasses to determine a preset number of groups of fourth pose information of the smart pen, wherein all the fourth pose information are the poses of the smart pen at the same time node;

[0096] Step D20, based on the weights assigned to the cameras respectively, all the fourth posture information are fused to generate fifth posture information, and a three-dimensional virtual image of the smart pen is constructed according to the fifth posture information.

[0097] In this embodiment, the preset number is at least two, and this embodiment is not limited. A preset number of groups of 6DoF cameras can be set on the smart glasses to obtain the fourth posture information generated by multiple 6DoF cameras tracking the smart pen, and the corresponding preset weight is assigned to the fourth posture information collected by each camera, and the fourth posture information of the preset weights corresponding to all cameras are summed to obtain the fifth posture information of the smart pen, and then the fifth posture information is used to construct the three-dimensional virtual image of the smart pen, wherein each of the above weight values ​​can be the same or different, and the sum of all weights can also be one, which is not limited in this embodiment. This embodiment collects multiple posture information of the smart pen at the same time, and determines the posture of the smart pen by integrating multiple posture information, thereby reducing errors and improving the accuracy of constructing the three-dimensional virtual image of the smart pen.

[0098] In another possible implementation, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:

[0099] Step E10, determining the distance between the pen tip of the three-dimensional virtual image and the virtual image interaction interface, and the change trend of the distance, wherein the change trend includes an increasing trend and a decreasing trend;

[0100] Step E20, if the distance matches the first preset distance, generating writing status information;

[0101] Step E30, if the distance matches the second preset distance, and the change trend is an increasing trend, then generating pen lifting state information;

[0102] Step E40, if the distance matches the second preset distance, and the change trend is a decreasing trend, then generating pen-down status information;

[0103] Step E50: If the distance matches a third preset distance, idle state information is generated, wherein the second preset distance is greater than the first preset distance, and the third preset distance is greater than the second preset distance.

[0104] In this embodiment, by determining the distance between the nib of the three-dimensional virtual image and the virtual image interaction interface, it can be determined whether the user is writing in the virtual space. The first preset distance is the distance at which the three-dimensional virtual image contacts the virtual interaction interface, which can be zero. The second preset distance is the distance between the three-dimensional virtual image and the virtual interaction interface when the three-dimensional virtual image is doing a pen-lifting action or a pen-putting action, and the third preset distance is the distance between the three-dimensional virtual image and the virtual interaction interface when the three-dimensional virtual image is not writing. Among them, the second preset distance is greater than the first preset distance, and the third preset distance is greater than the second preset distance. The values ​​can be set by technicians in this technical field. This embodiment can determine whether the user is writing in the virtual space by determining the distance between the nib of the smart pen and the virtual image interaction interface, and provides a judgment condition for real-time adjustment of the vibration characteristics of the nib of the smart pen.

[0105] Exemplarily, if the change trend of the above distance is an increasing trend, the 3D virtual image is moving away from the virtual interactive interface, and the above distance matches the second preset distance, it can be determined that the 3D virtual image is doing a pen-lifting action, and the smart glasses generate pen-lifting state information accordingly; if the change trend of the above distance is a decreasing trend, the 3D virtual image is approaching the virtual interactive interface, and the above distance matches the second preset distance, it can be determined that the 3D virtual image is doing a pen-dropping action, and the smart glasses generate pen-dropping state information accordingly. In this embodiment, when the 3D virtual image is doing a pen-dropping action, the vibration characteristics of the pen tip of the smart pen are slowly increased, for example, the vibration frequency and / or vibration amplitude of the pen tip of the smart pen are slowly increased. When the three-dimensional virtual image makes a pen-lifting action, the vibration characteristics of the pen tip of the smart pen are slowly reduced, for example, the vibration frequency and / or vibration amplitude of the pen tip of the smart pen are slowly reduced, so that the change process of the vibration characteristics is smoother and the sense of frustration caused by the sudden change of the vibration characteristics is reduced. This embodiment can determine whether the user is writing or lifting the pen in the virtual space by determining the changing trend of the distance between the pen tip of the three-dimensional virtual image and the virtual image interaction interface, thereby providing a judgment condition for real-time adjustment of the vibration characteristics of the pen tip of the smart pen.

[0106] In this embodiment, the smart glasses reconstruct a three-dimensional virtual image of the smart pen in the virtual space, so that the user can determine the writing position based on the three-dimensional virtual image, and can correctly find the position where writing was last paused when continuing to write, thereby improving the convenience of writing for users in the virtual space. In addition, this embodiment generates and sends usage status information to the smart pen based on the three-dimensional virtual image, thereby providing judgment conditions for real-time adjustment of the vibration characteristics of the pen tip of the smart pen, so that the smart pen can provide the user with real-time writing feedback based on the judgment conditions, thereby improving the user's writing efficiency in the virtual space.

[0107] Embodiment 3

[0108] An embodiment of the present invention provides a pair of smart glasses, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the writing method in the above-mentioned embodiment one.

[0109] Reference below Figure 6 , which shows a schematic diagram of the structure of smart glasses suitable for implementing the embodiments of the present disclosure. The smart glasses in the embodiments of the present disclosure include but are not limited to mixed reality (Mixed Reality) - MR glasses, augmented reality (Augmented Reality) - AR glasses, extended reality (Extended Reality) - XR glasses or some combination thereof. Figure 6 The smart glasses shown are merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 6 As shown, the smart glasses may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). In RAM 1004, various programs and data required for the operation of the smart glasses are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0111] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the smart glasses to communicate wirelessly or wired with other devices to exchange data. Although the figure shows smart glasses with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0113] The smart glasses provided by the present invention, using the writing method in the above embodiment, can solve the technical problem that it is impossible to truly and accurately provide real-time writing feedback to users, resulting in low writing efficiency of users in virtual space. Compared with the prior art, the beneficial effects of the smart glasses provided by the embodiment of the present invention are the same as the beneficial effects of the writing method provided by the above embodiment, and other technical features in the smart glasses are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0114] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0115] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0116] Embodiment 4

[0117] This embodiment also provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the writing method in the above embodiment.

[0118] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0119] The computer-readable storage medium may be included in a smart pen or smart glasses; or may exist independently without being assembled into a smart pen or smart glasses.

[0120] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the smart pen, the smart pen: establishes a communication connection link with the smart glasses; based on the communication connection link, dynamically receives the usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through the camera of the smart glasses; dynamically adjusts the vibration characteristics of the pen tip part of the smart pen according to the usage status information so that the vibration characteristics match the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

[0121] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the smart glasses, the smart glasses: after establishing a communication connection link with the smart glasses, track the features on the smart pen housing through the camera of the smart glasses to determine the first pose information of the smart pen; construct a three-dimensional virtual image of the smart pen according to the first pose information; generate usage status information of the smart pen according to the three-dimensional virtual image; and send the usage status information to the smart pen so that the smart pen dynamically adjusts the vibration characteristics of the pen tip part of the smart pen according to the usage status information.

[0122] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0125] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned writing method, which solves the technical problem of how to truly and accurately provide real-time writing feedback to users and improve the writing efficiency of users in virtual space. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the writing method provided by the above-mentioned embodiment, which will not be described in detail here.

[0126] Embodiment 5

[0127] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the writing method described above are implemented.

[0128] The computer program product provided by the present application can solve the technical problem that it is impossible to provide users with real-time writing feedback in a true and accurate manner, resulting in low writing efficiency of users in a virtual space. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the writing method provided by the above-mentioned embodiment 1 or embodiment 2, and will not be elaborated here.

[0129] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A writing method, characterized in that: Applied to smart pens, including: Establishing a communication link with the smart glasses; Based on the communication connection link, dynamically receive usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through a camera of the smart glasses; The vibration characteristics of the pen tip of the smart pen are dynamically adjusted according to the usage status information so that the vibration characteristics match the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

2. The writing method according to claim 1, characterized in that: The use state information includes one of idle state information, writing state information, pen-down state information and pen-up state information. The step of dynamically adjusting the vibration characteristics of the pen tip of the smart pen according to the use state information includes: If the usage state information is idle state information, setting the vibration characteristic to a first preset vibration characteristic; If the usage status information is writing status information, the vibration feature is set to a second preset vibration feature, wherein the vibration frequency of the second preset vibration feature is greater than the vibration frequency of the first preset vibration feature, and / or the vibration amplitude of the second preset vibration feature is greater than the vibration amplitude of the first preset vibration feature; If the use status information is the writing status information, gradually increasing the vibration characteristic from the first preset vibration characteristic to the second preset vibration characteristic within a preset time period; If the usage status information is pen-lifting status information, the vibration characteristic is gradually reduced from the second preset vibration characteristic to the first preset vibration characteristic within a preset time period.

3. The writing method according to claim 1, characterized in that: Before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes: Dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen; The second posture information is sent to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

4. The writing method according to claim 1, wherein: Before the step of establishing a communication connection link with the smart glasses, the method further includes: Acquire pressure parameters of the smart pen housing; If the pressure parameter is greater than a preset pressure parameter threshold, the step of establishing a communication connection link with the smart glasses is performed.

5. A writing method, characterized in that: Applied to smart glasses, including: After establishing a communication connection link with the smart glasses, tracking the features on the housing of the smart pen through the camera of the smart glasses to determine the first position information of the smart pen; constructing a three-dimensional virtual image of the smart pen according to the first posture information; generating usage status information of the smart pen according to the three-dimensional virtual image; The usage status information is sent to the smart pen, so that the smart pen dynamically adjusts the vibration characteristics of the pen tip part of the smart pen according to the usage status information, wherein the vibration characteristics include vibration frequency and vibration amplitude.

6. The writing method according to claim 5, characterized in that: The step of constructing the three-dimensional virtual image of the smart pen according to the first posture information comprises: Receiving second position information sent by the smart pen, wherein the first position information and the second position information are the position of the smart pen measured at the same time node; The first posture information of the first preset weight and the second posture information of the second preset weight are summed to obtain third posture information, and a three-dimensional virtual image of the smart pen is constructed according to the third posture information.

7. The writing method according to claim 5, characterized in that: The step of generating the usage status information of the smart pen according to the three-dimensional virtual image comprises: Determine the distance between the pen tip of the three-dimensional virtual image and the virtual image interaction interface, and the change trend of the distance, wherein the change trend includes an increasing trend and a decreasing trend; If the distance matches the first preset distance, generating writing status information; If the distance matches the second preset distance, and the change trend is an increasing trend, then generating pen lifting state information; If the distance matches the second preset distance, and the change trend is a decreasing trend, then generating pen-down state information; If the distance matches a third preset distance, idle state information is generated, wherein the second preset distance is greater than the first preset distance, and the third preset distance is greater than the second preset distance.

8. A smart pen, characterized in that: The smart pen comprises: a memory, a processor, and a writing program stored in the memory and executable on the processor, wherein the writing program is configured to implement the steps of the writing method according to any one of claims 1 to 4.

9. A smart glasses, characterized in that: The smart glasses include: a memory, a processor, and a writing program stored in the memory and executable on the processor, wherein the writing program is configured to implement the steps of the writing method as described in any one of claims 5 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a writing program, and when the writing program is executed by a processor, the steps of the writing method according to any one of claims 1 to 7 are implemented.

Images