Air writing method, intelligent pen, intelligent glasses and computer readable storage medium

Abstract

The invention discloses an aerial writing method, an intelligent pen, intelligent glasses and a computer readable storage medium, and belongs to the technical field of human-computer interaction. According to the invention, a communication connection link is established with the intelligent glasses; based on the communication connection link, receiving use state information of the intelligent pen sent by the intelligent glasses; if the use state information is writing state information, controlling an audio module of the intelligent pen to output a preset target audio signal; if the use state information is the idle state information, the audio module of the intelligent pen is controlled to stop outputting the preset target audio signal, so that real-time writing feedback is truly and accurately provided for the user, and the writing efficiency of the user in the virtual space is improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to an air writing method, a smart pen, smart glasses, and a computer-readable storage medium. Background Art

[0002] With the development of technologies such as virtual reality and augmented reality, users can obtain more and more complete information in virtual space. While obtaining information, they also need to input information. Compared with keyboard input, it is more efficient and convenient for users to input information using smart pens, which can meet the needs of a large number of text and painting information interaction scenarios. Users can use smart pens to write freely in virtual space.

[0003] Generally speaking, the virtual system needs the tip of the smart pen to touch the virtual interactive interface before it can recognize the information input by the user. However, when the user writes in the virtual space, the smart pen is used in the air, and the user cannot feel the real touch of the pen touching the real object. Therefore, the user does not know whether the tip of the smart pen touches the virtual interactive interface. This will bring about the problem of invalid writing. That is, the user waves the smart pen to write, but finds that the content has not been successfully entered in the virtual interactive interface. Therefore, the user needs to repeatedly adjust and re-enter the content, which seriously reduces the user's writing efficiency in the virtual space. Summary of the invention

[0004] The main purpose of this application is to provide an air writing method, a smart pen, smart glasses and a computer-readable storage medium, aiming to solve the technical problem that it is impossible to provide users with real-time writing feedback truthfully and accurately, resulting in low writing efficiency of users in virtual space.

[0005] To achieve the above objectives, the present application provides an air writing method, which is applied to a smart pen, comprising:

[0006] Establishing a communication link with the smart glasses;

[0007] Based on the communication connection link, receiving usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through a camera of the smart glasses, and the usage status information includes idle state information and writing state information;

[0008] If the usage status information is writing status information, controlling the audio module of the smart pen to output a preset target audio signal;

[0009] If the usage status information is idle status information, the audio module of the smart pen is controlled to stop outputting a preset target audio signal.

[0010] Optionally, before the step of controlling the audio module of the smart pen to output a preset target audio signal, the method further includes:

[0011] Receiving the virtual writing material currently written by the smart pen sent by the smart glasses;

[0012] According to the virtual writing material, querying from a preset audio feature mapping table to obtain a target audio feature of the virtual writing material mapping, wherein the target audio feature includes an audio phase feature and an audio amplitude feature;

[0013] The audio signal corresponding to the target audio feature is used as a preset target audio signal.

[0014] Optionally, before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:

[0015] Dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen;

[0016] The second posture information is sent to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

[0017] To achieve the above objectives, the present application provides an air writing method, which is applied to smart glasses and includes:

[0018] After establishing a communication connection link with the smart glasses, the first position information of the smart pen is determined by tracking the feature on the housing of the smart pen through the camera of the smart glasses;

[0019] constructing a three-dimensional virtual image of the smart pen according to the first posture information;

[0020] generating usage status information of the smart pen according to the three-dimensional virtual image;

[0021] The usage status information is sent to the smart pen, so that after the smart pen determines that the usage status information is writing status information, it triggers the audio module of the smart pen to output a preset target audio signal.

[0022] Optionally, the method further comprises:

[0023] Get the current interface configuration parameters of the virtual image interaction interface;

[0024] Determining the material of the virtual writing object currently written by the smart pen according to the current interface configuration parameters;

[0025] The virtual writing material is sent to the smart pen, so that the smart pen determines the target audio feature mapped by the virtual writing material according to the virtual writing material, and uses the audio signal corresponding to the target audio feature as a preset target audio signal.

[0026] Optionally, the step of constructing the three-dimensional virtual image of the smart pen according to the first posture information includes:

[0027] Receiving second position information sent by the smart pen, wherein the first position information and the second position information are the position of the smart pen measured at the same time node;

[0028] The first posture information of the first preset weight and the second posture information of the second preset weight are summed to obtain third posture information, and a three-dimensional virtual image of the smart pen is constructed according to the third posture information.

[0029] Optionally, the method further comprises:

[0030] Determining the distance between the pen tip portion of the three-dimensional virtual image and the virtual image interaction interface;

[0031] If the distance matches the first preset distance, generating writing status information;

[0032] If the distance matches a second preset distance, idle state information is generated, wherein the second preset distance is greater than the first preset distance.

[0033] The present application also provides a smart pen, which is a physical device. The smart pen includes: a memory, a processor, and a program of the air writing method stored in the memory and executable on the processor. When the program of the air writing method is executed by the processor, the steps of the air writing method described above can be implemented.

[0034] The present application also provides a pair of smart glasses, which is a physical device. The smart glasses include: a memory, a processor, and a program of the air writing method stored in the memory and executable on the processor. When the program of the air writing method is executed by the processor, the steps of the air writing method described above can be implemented.

[0035] The present application provides an air writing method, a smart pen, smart glasses and a computer-readable storage medium. By receiving the use status information of the smart pen sent by the smart glasses, wherein the use status information is generated by tracking the smart pen through the camera of the smart glasses, and the use status information includes idle state information and writing state information, if the use status information is writing state information, the audio module of the smart pen is controlled to output a preset target audio signal; if the use status information is idle state information, the audio module of the smart pen is controlled to stop outputting the preset target audio signal, thereby simulating the sound signal when the pen tip writes on the physical medium when the user writes on the physical medium, thereby giving the user auditory feedback. When the pen in the virtual world does not come into contact with the virtual interactive interface, the user can truly feel it and promptly and accurately drop the pen tip of the virtual world pen on the virtual interactive interface, instead of realizing the need to adjust the smart pen until the written content is found not to appear on the virtual interactive interface after writing for a period of time, thereby providing the user with real-time writing feedback truly and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space.

[0036] That is, when the tip of the smart pen is detected to reach the interactive input interface of the virtual image, the audio module of the smart glasses generates a sound signal that simulates the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby simulating the sound of the smart pen on the surface of the physical medium. When the tip of the smart pen leaves the interactive input interface of the virtual image, the sound stops being played, allowing the user to obtain a more realistic writing experience, reducing the discomfort of writing in the virtual world, and effectively avoiding invalid writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a system block diagram of the smart glasses involved in the air writing method of this application;

[0040] Figure 2 A system block diagram of the smart pen involved in the air writing method of this application;

[0041] Figure 3This is a schematic diagram of the flow chart of the first embodiment of the air writing method of the present application;

[0042] Figure 4 A schematic diagram of a scene for writing in the air according to an embodiment of the present application;

[0043] Figure 5 This is a flow chart of a second embodiment of the air writing method of the present application;

[0044] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the smart glasses in this embodiment.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] With the development of technologies such as virtual reality and augmented reality, users can obtain more and more complete information in virtual space. While obtaining information, they also need to input information. Compared with keyboard input, it is more efficient and convenient for users to input information using smart pens, which can meet the needs of a large number of text and painting information interaction scenarios. Users can use smart pens to write freely in virtual space.

[0048] Generally speaking, the virtual system needs the tip of the smart pen to touch the virtual interactive interface before it can recognize the information input by the user. However, when the user writes in the virtual space, the smart pen is used in the air, and the user cannot feel the real touch of the pen touching the real object. Therefore, the user does not know whether the tip of the smart pen touches the virtual interactive interface. This will bring about the problem of invalid writing. That is, the user waves the smart pen to write, but finds that the content has not been successfully entered in the virtual interactive interface. Therefore, the user needs to repeatedly adjust and re-enter the content, thereby reducing the user's writing efficiency in the virtual space.

[0049] The present application provides an air writing method, a smart pen, smart glasses and a computer-readable storage medium. By receiving the use status information of the smart pen sent by the smart glasses, wherein the use status information is generated by tracking the smart pen through the camera of the smart glasses, and the use status information includes idle state information and writing state information, if the use status information is writing state information, the audio module of the smart pen is controlled to output a preset target audio signal; if the use status information is idle state information, the audio module of the smart pen is controlled to stop outputting the preset target audio signal, thereby simulating the sound signal when the pen tip writes on the physical medium when the user writes on the physical medium, thereby giving the user auditory feedback. When the pen in the virtual world does not come into contact with the virtual interactive interface, the user can truly feel it and promptly and accurately drop the pen tip of the virtual world pen on the virtual interactive interface, instead of realizing the need to adjust the smart pen until the written content is found not to appear on the virtual interactive interface after writing for a period of time, thereby providing the user with real-time writing feedback truly and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space. That is, when the tip of the smart pen is detected to reach the interactive input interface of the virtual image, the audio module of the smart glasses generates a sound signal that simulates the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby effectively simulating the sound of the smart pen on the surface of the physical medium. When the tip of the smart pen leaves the interactive input interface of the virtual image, the sound stops playing, thereby allowing the user to obtain a more realistic writing experience, reducing the discomfort of writing in the virtual world, and effectively avoiding invalid writing.

[0050] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0051] Reference Figure 1 , Figure 1This is a block diagram of the smart glasses system structure involved in the embodiment of the present application. The smart glasses may include an application processor, such as a central processing unit (CPU), a wireless communication module, a graphics renderer, a 6DoF (Degrees of Freedom) camera module, a virtual image distance adjustment module, an audio module and a speaker. In addition, the smart glasses may also include a memory, which may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. Among them, the wireless communication module is used to interact with the smart pen for data, the graphics renderer is used to project the virtual image into the display module of the smart glasses, the virtual image distance adjustment module is used to adjust the display virtual image distance, and the 6DoF camera module is used to track the smart pen to obtain the posture information of the smart pen. In addition, the audio module is used to generate a sound signal that simulates the pen tip writing on the surface of a physical medium and plays it through a speaker, thereby simulating the sound of the smart pen writing on the surface of a physical medium. When the pen tip of the smart pen leaves the interactive input interface of the virtual image, the sound is stopped from being generated and played, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the virtual space, and improve the writing efficiency of the user in the virtual space. Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the smart glasses, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The above-mentioned smart glasses are also not limited to smart products such as AR (Augmented Reality) glasses, XR (Extended Reality) glasses, MR (Mixed Reality) glasses or other head-mounted display devices.

[0052] Reference Figure 2 , Figure 2This is a block diagram of the smart pen system structure involved in the embodiment of the present application. The smart pen may include a microprocessor, such as a central processing unit (CPU), a wireless communication module, an IMU (Inertial Measurement Unit) module, and an IR LED (Infrared Light Emitting Diode) array module. In addition, it may also include a memory, which may be a high-speed random access memory or a stable non-volatile memory, such as a disk storage. Among them, the wireless communication module is used to interact with the smart glasses for data, and the IMU module is used to collect the position and posture of the smart pen itself in real time. The IR LED array module may include a plurality of IR LED lights and drivers distributed three-dimensionally on the smart pen, so that the user can be recognized by the 6DOF camera module of the smart glasses when holding the pen in different postures or angles. It can also be replaced by a highly reflective graphic or feature object, so that the smart glasses can track the smart pen. Even in a dark environment, the 6DoF camera of the smart glasses can clearly identify the smart pen. Those skilled in the art can understand that Figure 2 The structure shown in the figure does not constitute a limitation to the smart pen, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] Embodiment 1

[0054] Reference Figure 3 The present application provides an air writing method. In a first embodiment of the air writing method, in this embodiment, the air writing method is applied to a smart pen, and the method includes the following steps:

[0055] Step S100, establishing a communication connection link with the smart glasses;

[0056] In this embodiment, the smart pen needs to exchange data with the smart glasses. Before the data exchange, the smart pen needs to establish a communication connection link with the smart glasses. The communication connection link can be a wireless connection. 。

[0057] Step S200, based on the communication connection link, receiving the usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through the camera of the smart glasses, and the usage status information includes idle state information and writing state information;

[0058] In this embodiment, the smart glasses can use a graphics renderer to project a virtual image on the display module, and build a virtual space with the help of the unity engine. The smart glasses will generate a virtual interactive interface in the virtual space, and at the same time, the smart pen will be reconstructed in three dimensions in the virtual space to generate a three-dimensional virtual image. That is to say, the motion posture of the smart pen in the real world and the motion posture of the three-dimensional virtual image of the smart pen in the virtual space are synchronized. When the user is ready to input content by handwriting, the virtual distance adjustment module will adjust the display virtual image from far to near. For example, when the display virtual image is located at a distance, it mainly provides a display function for the user. Therefore, when the display virtual image is at a distance, the distance from the smart glasses can be 2 meters, and when the display virtual image is located near, a virtual interactive interface will be generated to facilitate the user to write in the virtual space. At this time, the distance between the virtual interactive interface and the smart glasses can be 0.5 meters. The above distance can also be set to other values ​​to better match the most comfortable writing distance for the user. This embodiment does not make specific limitations.

[0059] In this embodiment, the smart glasses track the smart pen through a camera, thereby dynamically obtaining the position information of the smart pen, and generating a three-dimensional virtual image of the smart pen in the virtual space according to the position information. The use status information of the smart pen can be determined through the three-dimensional virtual image. The smart pen dynamically receives the use status information sent by the smart glasses through a communication connection link. The use status information can be one of writing status information and idle status information. When the tip of the three-dimensional virtual image touches the virtual interactive interface, the smart glasses can generate writing status information. If the use status information received by the smart pen is writing status information, it means that the user's writing content in the virtual space can be displayed on the virtual interactive interface in a timely manner. When the tip of the three-dimensional virtual image does not touch the virtual interactive interface and there is a certain distance from the virtual interactive interface, the smart glasses can generate idle status information. If the use status information received by the smart pen is idle status information, it means that the user's writing content in the virtual space cannot be displayed on the virtual interactive interface.

[0060] After step S200, step S300 is performed, and if the use status information is writing status information, the audio module of the smart pen is controlled to output a preset target audio signal;

[0061] Step S400: If the usage status information is idle status information, the audio module of the smart pen is controlled to stop outputting a preset target audio signal.

[0062] In this embodiment, if the usage status information received by the smart pen is writing status information, it indicates that the user is writing on the virtual interactive interface, or is preparing to write. In order to simulate the auditory experience of the user when writing on the physical medium in reality, the audio module of the smart glasses generates a sound signal that simulates the pen tip writing on the surface of the physical medium and plays it through the speaker, effectively simulating the sound of the smart pen on the surface of the physical medium (i.e., the target audio signal). If the usage status information received by the smart pen is idle status information, it indicates that the user stops writing on the virtual interactive interface, and the pen does not touch the virtual interactive interface in the virtual space. Therefore, in order to simulate the auditory experience of the user in this scenario, the sound (i.e., the target audio signal) is stopped from being played, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the void, and effectively avoid invalid writing.

[0063] In this embodiment, by receiving the usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through the camera of the smart glasses, and the usage status information includes idle status information and writing status information. If the usage status information is writing status information, the audio module of the smart pen is controlled to output a preset target audio signal; if the usage status information is idle status information, the audio module of the smart pen is controlled to stop outputting the preset target audio signal, thereby simulating the sound signal when the pen tip writes on the physical medium when the user writes on the physical medium, thereby giving the user auditory feedback. When the pen in the virtual world does not come into contact with the virtual interactive interface, the user can truly feel it and promptly and accurately drop the pen tip of the virtual world pen on the virtual interactive interface, instead of realizing the need to adjust the smart pen until the written content is found not to appear on the virtual interactive interface after writing for a period of time, thereby providing the user with real-time writing feedback truly and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space.

[0064] That is, in this embodiment, when it is detected that the tip of the smart pen reaches the interactive input interface of the virtual image, the audio module of the smart glasses generates a sound signal that simulates the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby simulating the sound of the smart pen on the surface of the physical medium. When the tip of the smart pen leaves the interactive input interface of the virtual image, the sound stops playing, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the virtual world, and effectively avoid invalid writing.

[0065] In a possible implementation, before the step of controlling the audio module of the smart pen to output a preset target audio signal, the method further includes:

[0066] Step A10, receiving the virtual writing material currently written by the smart pen sent by the smart glasses;

[0067] Step A20, according to the virtual writing material, querying from a preset audio feature mapping table to obtain a target audio feature of the virtual writing material mapping, wherein the target audio feature includes an audio phase feature and an audio amplitude feature;

[0068] Step A30: taking the audio signal corresponding to the target audio feature as a preset target audio signal.

[0069] In this embodiment, the virtual writing material refers to the material of the virtual object medium displayed in the interface of the smart glasses, and when the user writes on the virtual writing material, a writing sound matching the virtual writing material will be generated. For example, the virtual writing material can be a virtual paper material or a virtual glass material.

[0070] Exemplarily, the smart glasses can define the interactive input interface as surfaces of different materials according to user needs or scene requirements, and the image renderer will render different textures for the virtual interactive interface, so that the user can see different texture images of the virtual interactive interface in the display module of the smart glasses. The smart glasses preset sound signals that simulate and compare writing on the surfaces of multiple materials. When the virtual interactive interface is defined as the surface of material A, the audio module generates a sound signal a that simulates the pen tip writing on the surface of material A and plays it through the speaker. When the interactive input interface is defined as the surface of material B, the audio module generates a sound signal b that simulates the pen tip writing on the surface of material B and plays it through the speaker, so that the user can get a more realistic writing experience.

[0071] This embodiment obtains the target audio features of the material mapping of the virtual writing object from a preset audio feature mapping table according to the material of the virtual writing object currently written by the smart pen sent by the smart glasses, wherein the target audio features include audio phase features and audio amplitude features, and uses the audio signals corresponding to the target audio features as the preset target audio signals, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the virtual space, effectively avoid invalid writing, further ensure that the real-time writing feedback is provided to the user in a realistic and accurate manner, and improve the user's writing efficiency in the virtual space.

[0072] In a possible implementation manner, before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:

[0073] Step B10, dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen;

[0074] Those skilled in the art will know that the inertial sensor is an IMU (Inertial Measurement Unit) sensor.

[0075] Step B20, sending the second posture information to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

[0076] In this embodiment, an IMU module is installed on the smart pen. Through the inertial sensor of the IMU module, the second posture information of the smart pen itself can be dynamically detected, and the second posture information is sent to the smart glasses through the communication connection link. The smart glasses track the smart pen through the camera to obtain the first posture information of the smart pen, and then build a three-dimensional virtual image of the smart pen in combination with the second posture information sent by the smart pen. In this embodiment, the smart pen sends its own posture information to the smart glasses through the communication connection link, which provides a data basis for the smart glasses to build a three-dimensional virtual image of the smart pen.

[0077] In an practicable manner, before the step of establishing a communication connection link with the smart glasses, the method further includes:

[0078] Step C10, obtaining the pressure parameter of the smart pen housing;

[0079] Step C20: If the pressure parameter is greater than a preset pressure parameter threshold, executing the step of establishing a communication connection link with the smart glasses.

[0080] In this embodiment, it is necessary to determine whether the user is using the smart pen. After determining that the user is using the smart pen, the smart pen and the smart glasses establish a communication connection link. At this time, a pressure sensor can be added to the housing of the smart pen to detect the change of the pressure parameter at the housing of the smart pen in real time. The preset pressure parameter threshold is set by the technicians in this technical field to accurately determine whether the smart pen is being used. When the smart pen is picked up (held) by the user, the pressure parameter of the smart pen housing is greater than the preset pressure parameter threshold, then it can be determined that the user is using the smart pen, and the smart pen and the smart glasses establish a communication connection link. When the smart pen is put down (not held) by the user, the pressure value of the smart pen housing is less than or equal to the preset pressure parameter threshold, then it is determined that the smart pen is not in use, and the smart pen is disconnected from the smart glasses. Optionally, a button or switch can be installed on the smart pen. When the user needs to use the smart pen, the smart pen can be manually turned on to establish a communication connection link between the smart pen and the smart glasses. This embodiment automatically controls the communication connection state between the smart pen and the smart glasses, as well as the opening and closing of the smart pen, by detecting the pressure parameters of the smart pen housing, thereby reducing the power consumption of the smart pen while improving the convenience of writing for users in the virtual space.

[0081] In this embodiment, the user uses a smart pen in a virtual space, and the smart glasses generate usage status information of the smart pen according to the three-dimensional virtual image of the smart pen in the virtual space, and share it with the smart pen in real time. The smart pen dynamically adjusts the vibration frequency of the pen tip according to the corresponding usage status information, so that the user can feel the touch of the pen when it contacts the physical medium when writing in the virtual world. The user can accurately judge whether he has performed effective writing based on the change in touch, and promptly and accurately place the pen tip of the virtual world on the virtual interactive interface. Therefore, this embodiment realizes the real-time and accurate provision of real-time writing feedback to the user, thereby improving the user's writing efficiency in the virtual space.

[0082] Embodiment 2

[0083] Please refer to Figure 5 The present application also proposes a second embodiment of the air writing method. In this embodiment, the air writing method is applied to smart glasses, and the method includes the following steps:

[0084] Step S400, after establishing a communication connection link with the smart glasses, tracking features on the housing of the smart pen through the camera of the smart glasses to determine the first position information of the smart pen;

[0085] Step S500, constructing a three-dimensional virtual image of the smart pen according to the first posture information;

[0086] Step S600, generating usage status information of the smart pen according to the three-dimensional virtual image;

[0087] Step S700: sending the usage status information to the smart pen, so that after the smart pen determines that the usage status information is writing status information, it triggers the audio module of the smart pen to output a preset target audio signal.

[0088] In this embodiment, the feature object may be an IR LED array module on the housing of the smart pen. The IR LED array module may include a plurality of IR LED lights and drivers that are three-dimensionally distributed on the smart pen, so that the 6DOF camera module of the smart glasses can recognize the feature object when the user holds the pen in different postures or at different angles. In addition, a highly reflective pattern may be used as a substitute, so that the smart glasses can track the smart pen, and the 6DoF camera of the smart glasses can clearly recognize the smart pen even in a dark environment. The first position information includes the real-time position of the smart pen measured by the smart glasses. The smart glasses track the smart pen through the 6DoF camera module, take photos or videos of the smart pen, and then calculate the first position information of the smart pen through the SLAM (Simultaneous Localization and Mapping) algorithm. The smart glasses may first calibrate the time and position of the 6DoF camera module and the display module in their own display coordinate system. Initially, the position of the three-dimensional virtual image of the smart pen in the display coordinate system may be the same as that of the display module. The smart glasses obtain the first position information of the smart pen, obtain the change of the position and posture of the smart pen according to the information, and reconstruct the three-dimensional virtual image of the smart pen. When a user wears smart glasses or other head-mounted display devices, he may not be able to see the position of the pen, which makes it difficult for the user to write. In one scenario, the user is writing content on a virtual interactive interface. For some reason, the user pauses writing. When the user wants to continue writing, he may not be able to find the position where he paused writing last time. The three-dimensional virtual image of the smart pen is displayed in real time in the virtual space, and the position of the three-dimensional virtual image can be prompted.

[0089] When the smart pen moves and changes its posture in the real world, the corresponding three-dimensional virtual image will also move in the display coordinate system and undergo the same posture change. At this time, the usage status information of the smart pen can be determined by calculating the displacement of the three-dimensional virtual image pen tip. Preferably, the usage status information of the smart pen can be determined by calculating the distance between the three-dimensional virtual image pen tip and the virtual interactive interface, and then the usage status information is sent to the smart pen. After the smart pen determines that the usage status information is writing status information, the smart pen triggers the audio module of the smart pen to output a preset target audio signal. After determining that the usage status information is idle status information, the smart pen triggers the audio module of the smart pen to stop outputting the preset target audio signal.

[0090] This embodiment obtains the posture information of the smart pen in real time, reconstructs the smart pen in the virtual space, and enables the user to accurately determine the writing position through the three-dimensional virtual image of the smart pen, thereby improving the convenience of the user's writing in the virtual world, and by sending the usage status information of the smart pen to the smart pen, after the smart pen determines that the usage status information is writing status information, the smart pen triggers the audio module of the smart pen to output a preset target audio signal, and after determining that the usage status information is idle status information, the smart pen triggers the audio module of the smart pen to stop outputting the preset target audio signal, thereby providing the user with real-time writing feedback truthfully and accurately, effectively avoiding invalid writing, reducing the number of times the user repeatedly inputs content, and thereby improving the user's writing efficiency in the virtual space.

[0091] That is, in the embodiment of the present application, when it is detected that the tip of the smart pen reaches the interactive input interface of the virtual image, the audio module of the smart glasses generates a sound signal that simulates the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby simulating the sound of the smart pen on the surface of the physical medium. When the tip of the smart pen leaves the interactive input interface of the virtual image, the sound stops being played, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the virtual world, and effectively avoid invalid writing.

[0092] In one practicable manner, the method further comprises:

[0093] Step D10, obtaining current interface configuration parameters of the virtual image interaction interface;

[0094] Step D20, determining the material of the virtual writing object currently written by the smart pen according to the current interface configuration parameters;

[0095] Step D30, sending the virtual writing material to the smart pen, so that the smart pen determines the target audio feature mapped by the virtual writing material according to the virtual writing material, and uses the audio signal corresponding to the target audio feature as the preset target audio signal.

[0096] In this embodiment, the virtual writing material refers to the material of the virtual object medium displayed in the interface of the smart glasses, and when the user writes on the virtual writing material, a writing sound matching the virtual writing material will be generated. For example, the virtual writing material can be a virtual paper material or a virtual glass material.

[0097] In this embodiment, the current interface configuration parameter may be a texture image of the virtual writing object, and the material of the virtual writing object may be determined according to the texture image of the virtual writing object. For example, the texture image of the virtual writing object may be a paper texture or a glass texture.

[0098] This embodiment determines the material of the virtual writing object currently written by the smart pen according to the current interface configuration parameters, and sends the virtual writing object material to the smart pen, so that the smart pen determines the target audio features mapped by the virtual writing object material according to the virtual writing object material, and uses the audio signal corresponding to the target audio feature as the preset target audio signal, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the virtual space, effectively avoid invalid writing, further ensure that the real-time writing feedback is provided to the user in a realistic and accurate manner, and improve the user's writing efficiency in the virtual space.

[0099] In order to help understand the technical concept or technical principle of this application, an example is given. In this example, please refer to Figure 4 , Figure 4 Schematic diagram of the scene of writing in the air according to the embodiment of the present application. When the tip of the smart pen in Unity reaches the interactive input interface of the virtual image, the audio module of the smart glasses generates a sound signal simulating the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby simulating the sound of the smart pen on the surface of the physical medium. When the tip of the smart pen in Unity penetrates the interactive input interface (i.e., the direction of penetration is away from the user), and within the distance range of △x, the audio module still generates a sound signal simulating the pen tip writing on the surface of the physical medium and plays it through the speaker, that is, the coordinates of the pen tip are on the interactive input interface or through the interactive input interface, and the audio module continues to generate a sound signal simulating the pen tip writing on the surface of the physical medium and plays it through the speaker, thereby simulating the sound of the smart pen continuously writing on the surface of the physical medium. When the tip of the smart pen leaves the interactive input interface of the virtual image (i.e., the direction of departure is toward the user), the sound is stopped from being generated and played, so that the user can obtain a more realistic writing experience, reduce the discomfort of writing in the void, effectively avoid invalid writing, and thus improve the writing efficiency of the user in the virtual space.

[0100] In a possible implementation, the step of constructing the three-dimensional virtual image of the smart pen according to the first posture information includes:

[0101] Step E10, receiving second posture information sent by the smart pen, wherein the first posture information and the second posture information are postures of the smart pen measured at the same time node;

[0102] Step E20, calculating the sum of the first pose information of the first preset weight and the second pose information of the second preset weight to obtain third pose information, and constructing a three-dimensional virtual image of the smart pen according to the third pose information.

[0103] In this embodiment, the second posture information is the posture information of the smart pen collected in real time by the IMU module, and is sent to the smart glasses after collection. The first posture information and the second posture information respectively include the posture of the smart pen measured by the smart glasses and the smart pen at the same time node. Since the first posture information and the second posture information may have certain errors, the two groups of posture information can be combined for comprehensive judgment at this time. Preset weights can be set for the two groups of postures respectively, and the first posture information of the first preset weight and the second posture information of the second preset weight are summed to calculate the third posture information, and then the third posture information is used to construct the three-dimensional virtual image of the smart pen. In this embodiment, the first preset weight and the second preset weight can be the same or different, and the sum of the first preset weight and the second preset weight can be one. This embodiment receives the second posture information measured by the smart pen itself, and determines the posture of the smart pen by combining the first posture information and the second posture information, thereby reducing the error and improving the accuracy of the construction of the three-dimensional virtual image of the smart pen.

[0104] In another possible implementation, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:

[0105] Step F10, simultaneously tracking the features on the housing of the smart pen by using a preset number of cameras on the smart glasses to determine a preset number of groups of fourth pose information of the smart pen, wherein all the fourth pose information are the poses of the smart pen at the same time node;

[0106] Step F20, fusing all the fourth posture information based on the weights assigned to the cameras, generating fifth posture information, and constructing a three-dimensional virtual image of the smart pen according to the fifth posture information.

[0107] In this embodiment, the preset number is at least two, and this embodiment is not limited. A preset number of groups of 6DoF cameras can be set on the smart glasses to obtain the fourth posture information generated by multiple 6DoF cameras tracking the smart pen, and the corresponding preset weight is assigned to the fourth posture information collected by each camera, and the fourth posture information of the preset weights corresponding to all cameras are summed to obtain the fifth posture information of the smart pen, and then the fifth posture information is used to construct the three-dimensional virtual image of the smart pen, wherein each of the above weight values ​​can be the same or different, and the sum of all weights can also be one, which is not limited in this embodiment. This embodiment collects multiple posture information of the smart pen at the same time, and determines the posture of the smart pen by integrating multiple posture information, thereby reducing errors and improving the accuracy of constructing the three-dimensional virtual image of the smart pen.

[0108] In another possible implementation, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:

[0109] Step G10, determining the distance between the pen tip of the three-dimensional virtual image and the virtual image interaction interface;

[0110] Step G20, if the distance matches the first preset distance, generating writing status information;

[0111] Step G30: If the distance matches a second preset distance, generating idle state information, wherein the second preset distance is greater than the first preset distance.

[0112] In this embodiment, by determining the distance between the tip of the three-dimensional virtual image and the virtual image interaction interface, it can be determined whether the user is writing in the virtual space. The first preset distance is the distance at which the three-dimensional virtual image contacts the virtual interaction interface, which can be zero. The second preset distance is the distance between the three-dimensional virtual image and the virtual interaction interface when the three-dimensional virtual image is not written. Among them, the second preset distance is greater than the first preset distance, and the value can be set by a technician in this technical field. This embodiment can determine whether the user is writing in the virtual space by determining the distance between the tip of the smart pen and the virtual image interaction interface, which provides a judgment condition for real-time control of whether the audio module of the smart glasses outputs a preset target audio signal.

[0113] In this embodiment, the smart glasses reconstruct a three-dimensional virtual image of the smart pen in the virtual space, so that the user can determine the writing position based on the three-dimensional virtual image, and can correctly find the position where writing was last paused when continuing to write, thereby improving the convenience of users writing in the virtual space. In addition, this embodiment generates and sends usage status information to the smart pen based on the three-dimensional virtual image, thereby providing a judgment condition for real-time control of whether the audio module of the smart glasses outputs a preset target audio signal, so that the smart pen can provide the user with real-time writing feedback based on the judgment condition, truly and accurately, thereby improving the user's writing efficiency in the virtual space.

[0114] Embodiment 3

[0115] An embodiment of the present invention provides a pair of smart glasses, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the air writing method in the above-mentioned embodiment 1.

[0116] Reference below Figure 6, which shows a schematic diagram of the structure of smart glasses suitable for implementing the embodiments of the present disclosure. The smart glasses in the embodiments of the present disclosure include but are not limited to mixed reality (Mixed Reality) - MR glasses, augmented reality (Augmented Reality) - AR glasses, extended reality (Extended Reality) - XR glasses or some combination thereof. Figure 6 The smart glasses shown are merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0117] like Figure 6 As shown, the smart glasses may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). In RAM 1004, various programs and data required for the operation of the smart glasses are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0118] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the smart glasses to communicate wirelessly or wired with other devices to exchange data. Although the figure shows smart glasses with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0119] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0120] The smart glasses provided by the present invention adopt the air writing method in the above embodiment, which can solve the technical problem that it is impossible to provide users with real-time writing feedback truthfully and accurately, resulting in low writing efficiency of users in virtual space. Compared with the prior art, the beneficial effects of the smart glasses provided by the embodiment of the present invention are the same as the beneficial effects of the air writing method provided by the above embodiment, and other technical features in the smart glasses are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0121] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0122] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0123] Embodiment 4

[0124] This embodiment also provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the air writing method in the above embodiment.

[0125] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0126] The computer-readable storage medium may be included in a smart pen or smart glasses; or may exist independently without being assembled into a smart pen or smart glasses.

[0127] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the smart pen, the smart pen: establishes a communication connection link with the smart glasses; based on the communication connection link, dynamically receives the usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through the camera of the smart glasses; and dynamically adjusts the vibration frequency of the pen tip part of the smart pen according to the usage status information so that the vibration frequency matches the usage status information.

[0128] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the smart glasses, the smart glasses are enabled to: establish a communication connection link with the smart glasses; based on the communication connection link, receive the usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through the camera of the smart glasses, and the usage status information includes idle status information and writing status information; if the usage status information is writing status information, the audio module of the smart pen is controlled to output a preset target audio signal; if the usage status information is idle status information, the audio module of the smart pen is controlled to stop outputting the preset target audio signal.

[0129] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0132] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned air writing method, which solves the technical problem of how to truly and accurately provide real-time writing feedback to users and improve the writing efficiency of users in virtual space. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the air writing method provided by the above-mentioned embodiment, which will not be repeated here.

[0133] Embodiment 5

[0134] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned air writing method are implemented.

[0135] The computer program product provided by the present application can solve the technical problem that it is impossible to provide users with real-time writing feedback in a real and accurate manner, resulting in low writing efficiency of users in virtual space. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the air writing method provided by the above-mentioned embodiment 1 or embodiment 2, and will not be described in detail here.

[0136] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A method for writing in the air, characterized in that: Applied to smart pens, including: Establishing a communication link with the smart glasses; Based on the communication connection link, receiving usage status information of the smart pen sent by the smart glasses, wherein the usage status information is generated by tracking the smart pen through a camera of the smart glasses, and the usage status information includes idle state information and writing state information; If the usage status information is writing status information, controlling the audio module of the smart pen to output a preset target audio signal; If the usage status information is idle status information, the audio module of the smart pen is controlled to stop outputting a preset target audio signal.

2. The air writing method according to claim 1, characterized in that: Before the step of controlling the audio module of the smart pen to output a preset target audio signal, the method further includes: Receiving the virtual writing material currently written by the smart pen sent by the smart glasses; According to the virtual writing material, querying from a preset audio feature mapping table to obtain a target audio feature of the virtual writing material mapping, wherein the target audio feature includes an audio phase feature and an audio amplitude feature; The audio signal corresponding to the target audio feature is used as a preset target audio signal.

3. The air writing method according to claim 1, characterized in that: Before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes: Dynamically detecting second posture information of the smart pen through an inertial sensor of the smart pen; The second posture information is sent to the smart glasses based on the communication connection link, so that the smart glasses respond to the second posture information and generate the usage status information of the smart pen in combination with the first posture information determined by the camera tracking the smart pen.

4. A method for writing in the air, characterized in that: Applied to smart glasses, including: After establishing a communication connection link with the smart glasses, the first position information of the smart pen is determined by tracking the feature on the housing of the smart pen through the camera of the smart glasses; constructing a three-dimensional virtual image of the smart pen according to the first posture information; generating usage status information of the smart pen according to the three-dimensional virtual image; The usage status information is sent to the smart pen, so that after the smart pen determines that the usage status information is writing status information, it triggers the audio module of the smart pen to output a preset target audio signal.

5. The air writing method according to claim 4, characterized in that: The method further comprises: Get the current interface configuration parameters of the virtual image interaction interface; Determining the material of the virtual writing object currently written by the smart pen according to the current interface configuration parameters; The virtual writing material is sent to the smart pen, so that the smart pen determines the target audio feature mapped by the virtual writing material according to the virtual writing material, and uses the audio signal corresponding to the target audio feature as a preset target audio signal.

6. The air writing method according to claim 4, characterized in that: The step of constructing the three-dimensional virtual image of the smart pen according to the first posture information comprises: Receiving second position information sent by the smart pen, wherein the first position information and the second position information are the position of the smart pen measured at the same time node; The first posture information of the first preset weight and the second posture information of the second preset weight are summed to obtain third posture information, and a three-dimensional virtual image of the smart pen is constructed according to the third posture information.

7. The air writing method according to claim 4, characterized in that: The method further comprises: Determining the distance between the pen tip portion of the three-dimensional virtual image and the virtual image interaction interface; If the distance matches the first preset distance, generating writing status information; If the distance matches a second preset distance, idle state information is generated, wherein the second preset distance is greater than the first preset distance.

8. A smart pen, characterized in that: The smart pen comprises: a memory, a processor, and an air writing program stored in the memory and executable on the processor, wherein the air writing program is configured to implement the steps of the air writing method according to any one of claims 1 to 3.

9. A smart glasses, characterized in that: The smart glasses include: a memory, a processor, and an air writing program stored in the memory and executable on the processor, wherein the air writing program is configured to implement the steps of the air writing method according to any one of claims 4 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an air writing program, and when the air writing program is executed by a processor, the steps of the air writing method according to any one of claims 1 to 7 are implemented.

Images