Head-mounted display equipment interaction method and device, interaction module and interaction pen

Abstract

The invention discloses a head-mounted display device interaction method and device, an interaction module and an interaction pen, and relates to the technical field of head-mounted display devices, the method is applied to the interaction module, the interaction module is installed on a target pen, and the method comprises the following steps: responding to an interaction event; determining an intersection condition of a virtual ray emitted from a pen point of a target pen and a virtual picture displayed by the head-mounted display equipment; when the intersection condition is that an intersection point exists between the virtual ray and the virtual picture, determining an intersection point coordinate of the intersection point, and sending an interaction event and the intersection point coordinate to the head-mounted display device, so that the head-mounted display device executes a first control operation based on the intersection point and the interaction event; and when the intersection condition is that the virtual ray and the virtual picture do not have the intersection point, sending the interaction event to the head-mounted display device, so that the head-mounted display device executes a second control operation based on the interaction event. The efficiency of interaction with the head-mounted display device in a teaching scene can be improved.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and in particular to an interaction method, device, interaction module and interaction pen for a head-mounted display device. Background Technology

[0002] With the rapid development of AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), and XR (Extended Reality) technologies, head-mounted display devices, including AR / VR / MR / XR devices, have gradually integrated into people's daily lives and become an important part of modern technological experiences. Among these, using head-mounted displays to assist users in learning has become an important application scenario for head-mounted displays.

[0003] Currently, in educational settings, users typically need to use pens for writing or drawing to promote coherent thinking and enhance knowledge internalization. However, common head-mounted displays (HUDs) used in educational settings are usually equipped with dedicated interactive devices, such as controllers and remote controls, to allow users to control the HUD. This means that when a user needs to control the HUD during a lesson, they must first put down their pen and then pick up the dedicated interactive device. This frequent switching leads to low operational efficiency of HUDs in educational settings, thus impacting the user's learning outcomes.

[0004] Therefore, improving the efficiency of interaction with head-mounted display devices in teaching scenarios is an urgent problem that needs to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide an interaction method, device, interaction module and interaction pen for head-mounted display devices, which aims to improve the efficiency of interaction with head-mounted display devices in teaching scenarios.

[0006] To achieve the above objectives, this application provides a head-mounted display device interaction method, wherein the head-mounted display device interaction method is applied to an interaction module, the interaction module is mounted on a target pen, and the head-mounted display device interaction method includes:

[0007] In response to an interactive event, determine the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device;

[0008] In the case where the virtual ray and the virtual screen intersect, the coordinates of the intersection point are determined, and the interaction event and the intersection point coordinates are sent to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event.

[0009] If the virtual ray and the virtual screen do not intersect, the interaction event is sent to the head-mounted display device so that the head-mounted display device can perform a second control operation based on the interaction event.

[0010] In one embodiment, the step of determining the intersection of a virtual ray emitted from the tip of the target pen with a virtual image displayed on the head-mounted display device includes:

[0011] The first pose information of the target pen is determined, and the starting point and emission direction of the virtual ray emitted from the tip of the target pen are determined based on the first pose information. The starting point coordinates of the starting point in the preset spatial coordinate system are also determined.

[0012] The second pose information of the head-mounted display device is obtained, and the position of the virtual image displayed by the head-mounted display device in the preset spatial coordinate system is determined based on the second pose information.

[0013] Based on the starting point coordinates, the emission direction, and the image position, the intersection of the virtual ray and the virtual image is determined.

[0014] In one embodiment, the interaction module includes a sensor module, and prior to the step of determining the intersection of a virtual ray emitted from the tip of the target pen with a virtual image displayed on the head-mounted display device in response to an interaction event, the method further includes:

[0015] Using a preset reference surface as the target reference surface, the sensor module monitors the posture of the target pen to determine the angle between the pen tip pointing direction and the target reference surface, wherein the angle is an acute angle.

[0016] When the included angle is less than a preset angle, the ray interaction mode is activated to emit a virtual ray from the tip of the target pen.

[0017] In one embodiment, the method further includes:

[0018] In response to a user-triggered calibration event, the plane where the target pen is located is determined as the calibration plane, the calibration plane is used as the target reference plane, and the process returns to the step of monitoring the posture of the target pen through the sensor module and determining the angle between the pen tip pointing direction and the target reference plane.

[0019] In one embodiment, the interaction event includes a gesture interaction event, and before the step of determining a preset interaction type corresponding to the interaction event in response to the interaction event, the method further includes:

[0020] The sensor module monitors the pose change data of the target pen.

[0021] The pose change data is matched with preset gesture data, and the gesture interaction event is triggered when the pose change data and the preset gesture data are successfully matched.

[0022] In one embodiment, the interaction module further includes buttons, and the interaction event further includes button interaction events. Before the step of determining a preset interaction type corresponding to the interaction event in response to the interaction event, the method further includes:

[0023] When a preset operation command triggered by the button is detected, the button interaction event is triggered.

[0024] In one embodiment, the method further includes:

[0025] The sensor module of the interaction module monitors the motion state of the target pen. If the duration of the target pen being in a stationary state or in an abnormal motion state exceeds a preset duration, a low-power sleep mode is activated until a wake-up command is received.

[0026] Specifically, when the target pen is in the abnormal motion state, the angle change rate of the target pen is greater than a first preset threshold and / or the acceleration of the target pen is greater than a second preset threshold.

[0027] Furthermore, to achieve the above objectives, this application also provides a head-mounted display device interaction device, which is applied to an interaction module, the interaction module being mounted on a target pen, and the head-mounted display device interaction device comprising:

[0028] A response module is used to respond to interactive events and determine the intersection of a virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device.

[0029] A directional interaction module is used to determine the intersection coordinates of the virtual ray and the virtual screen when the intersection situation is that the virtual ray and the virtual screen have an intersection point, and send the interaction event and the intersection coordinates to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event;

[0030] A non-directional interaction module is used to send the interaction event to the head-mounted display device when the virtual ray and the virtual screen do not intersect, so that the head-mounted display device can perform a second control operation based on the interaction event.

[0031] In addition, to achieve the above objectives, this application also provides an interactive pen, which includes an interactive module.

[0032] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, storing a program that implements a head-mounted display device interaction method. The program that implements the head-mounted display device interaction method is executed by a processor to implement the steps of the head-mounted display device interaction method as described above.

[0033] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the head-mounted display device interaction method described above.

[0034] This application provides an interaction method for a head-mounted display device. The method is applied to an interaction module mounted on a target pen and communicatively connected to the head-mounted display device. In response to an interaction event, this application determines the intersection of a virtual ray emitted from the pen tip with a virtual image displayed on the head-mounted display device. If an intersection point exists, the coordinates of the intersection point are determined, and the interaction event and the coordinates are sent to the head-mounted display device, allowing the device to perform a corresponding first control operation based on the intersection point and the interaction event. If no intersection point exists, the interaction event is sent to the head-mounted display device, allowing it to perform a corresponding second control operation based on the interaction event.

[0035] In summary, compared to traditional methods that require specific interactive devices to interact with head-mounted displays, this application designs an interactive module that communicates with a head-mounted display. This module can be installed on a pen used by the user in a teaching scenario. Specifically, the interactive module responds to interactive events and determines the intersection of a virtual ray emitted from the pen tip with the virtual image displayed on the head-mounted display. If the intersection is found, the interactive event and the intersection coordinates are sent to the head-mounted display, allowing it to perform a first control operation based on the intersection and the interactive event. If not, the interactive event is directly sent to the head-mounted display, allowing it to perform a second control operation. Thus, the interactive module enables both directional and non-directional interaction with the head-mounted display. In other words, the pen equipped with this interactive module simultaneously possesses interactive and writing functions, avoiding the need for frequent switching between the pen and interactive device in teaching scenarios. This improves the efficiency of user interaction with the head-mounted display in teaching scenarios, ensures the continuity of user learning, and ultimately enhances the learning effect based on the head-mounted display. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the first embodiment of the head-mounted display device interaction method of this application;

[0039] Figure 2 This is a schematic diagram of the interactive pen architecture involved in an embodiment of the head-mounted display device interaction method of this application;

[0040] Figure 3 This is a schematic diagram of a calibration operation involved in an embodiment of the head-mounted display device interaction method of this application;

[0041] Figure 4 This is a schematic diagram of the module structure of the interactive device for the head-mounted display device of this application;

[0042] Figure 5 This is a schematic diagram of the hardware operating environment involved in the head-mounted display device interaction method in the embodiments of this application.

[0043] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0045] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The main solution of this application is as follows: In response to an interaction event, determine the intersection of a virtual ray emitted from the tip of the target pen with a virtual image displayed on the head-mounted display device; if the virtual ray and the virtual image intersect at a point, determine the coordinates of the intersection point, and send the interaction event and the coordinates of the intersection point to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event; if the virtual ray and the virtual image do not intersect at a point, send the interaction event to the head-mounted display device so that the head-mounted display device can perform a second control operation based on the interaction event.

[0047] Currently, in educational settings, users typically need to use pens for writing or drawing to promote coherent thinking and enhance knowledge internalization. However, common head-mounted displays (HUDs) used in educational settings are usually equipped with dedicated interactive devices, such as controllers and remote controls, to allow users to control the HUD. This means that when a user needs to control the HUD during a lesson, they must first put down their pen and then pick up the dedicated interactive device. This frequent switching leads to low operational efficiency of HUDs in educational settings, thus impacting the user's learning outcomes.

[0048] This application designs an interactive module that communicates with a head-mounted display device. This module can be installed on a pen used by the user in a teaching scenario. Specifically, the interactive module responds to interactive events by determining the intersection of a virtual ray emitted from the pen tip with the virtual image displayed on the head-mounted display device. If the intersection is found, the interactive event and the intersection coordinates are sent to the head-mounted display device, allowing the device to perform a first control operation based on the intersection and the interactive event. If not, the interactive event is directly sent to the head-mounted display device, allowing it to perform a second control operation. In other words, the interactive module enables both directional and non-directional interaction with the head-mounted display device. This means that the pen equipped with the interactive module simultaneously possesses interactive and writing functions, avoiding the need for frequent switching between the pen and the interactive device in teaching scenarios. This improves the efficiency of user interaction with the head-mounted display device in teaching scenarios, ensures the continuity of user learning, and ultimately enhances the learning effect based on the head-mounted display device.

[0049] It should be noted that the execution subject of the method in various embodiments of the head-mounted display device interaction method of this application can be an interaction module, and this embodiment does not specifically limit this. The following uses the interaction module as the execution subject to describe this embodiment and the following embodiments.

[0050] Based on this, this application proposes a head-mounted display device interaction method according to a first embodiment. The head-mounted display device interaction method is applied to an interaction module, which is mounted on a target pen and is communicatively connected to the head-mounted display device. Please refer to... Figure 1 The interaction method of the head-mounted display device includes steps S10 to S30:

[0051] Step S10: In response to an interactive event, determine the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device.

[0052] The head-mounted display devices in this embodiment include, but are not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof, and other display devices with virtual reality functions.

[0053] It should be noted that the interaction module is a collection of multiple functional modules. The interaction module includes a communication module for communicating with the head-mounted display device. In this embodiment, a low-power Bluetooth communication module is used as the communication module, thus improving battery life by employing BLE (Bluetooth Low Energy) mode. The interaction module also includes a charging module, which powers the interaction module through wireless charging or by replacing a button battery. The target pen refers to a conventional pen used by the user in the current teaching scenario, such as a ballpoint pen, pencil, or fountain pen. The interaction module can be a component that can be installed and fixed on various types of pens. For example, the interaction module can be a rigid pen sleeve or cap, meaning the interaction module is installed on the pen in the form of a pen sleeve or cap. The interaction module is detachable and versatile. Furthermore, the interaction module can detect the tip position of the target pen and emit a virtual ray from that tip position. When the virtual ray points to the virtual image displayed on the head-mounted display device, it can serve a teaching function; for example, the content pointed to by the virtual ray is marked, or the landing point of the virtual ray on the virtual image becomes a cursor.

[0054] In response to user interaction events triggered by the interaction module, determine the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device.

[0055] In one feasible implementation, step S10 may include:

[0056] Step S101: Determine the first pose information of the target pen, determine the starting point and emission direction of the virtual ray emitted from the tip of the target pen based on the first pose information, and determine the starting point coordinates of the starting point in a preset spatial coordinate system.

[0057] It should be noted that a spatial coordinate system for the current teaching scenario is pre-established. For example, the origin of the spatial coordinate system can be used as the location of the head-mounted display device when it is turned on. The interactive module of this application includes a sensor module, which includes, but is not limited to, accelerometers, gyroscopes, and magnetometers. This sensor module is also known as a 3DoF (Three Degrees of Freedom) hardware module. It should be understood that this 3DoF hardware module is typically very small, on the millimeter scale, and therefore can be placed within the interactive module, such as... Figure 2The diagram shows the architecture of the interactive pen. Area 1 (marked) represents the interactive module (in the form of a pen sleeve) surrounding the regular pen, and area 2 (marked) indicates the location of the 3DoF hardware module within the interactive module. This area can be understood as the gripping part of the pen, ensuring precise user operation. The target pen with the interactive module installed is referred to as the interactive pen. The sensor module is used to monitor the pose of the target pen. The interactive module in this application also includes an MCU (Microcontroller Unit) for processing the sensor data monitored by the sensor module to obtain the pose information of the target pen.

[0058] The sensor module monitors the pose of the target pen to obtain sensor data, and the MCU determines the pose information of the target pen (hereinafter referred to as the first pose information for distinction) based on this sensor data. It should be noted that the first pose information refers to the pose information of the target pen when an interactive event is received. Then, based on the first pose information, the starting point and emission direction of the virtual ray emitted from the tip of the target pen are determined, and the coordinates of the starting point in a preset spatial coordinate system are determined (hereinafter referred to as the starting point coordinates for distinction).

[0059] Step S102: Obtain the second pose information of the head-mounted display device, and determine the position of the virtual image displayed by the head-mounted display device in the preset spatial coordinate system based on the second pose information;

[0060] It should be noted that the interaction module communicates with the head-mounted display device via a communication module. The head-mounted display device also has a pose monitoring module to monitor its pose. When the interaction module receives an interaction event, it sends a pose request command to the head-mounted display device to request the current pose information of the head-mounted display device (hereinafter referred to as the second pose information for distinction).

[0061] The second pose information of the head-mounted display device is obtained, and the position of the virtual image displayed by the head-mounted display device in the preset spatial coordinate system is determined based on the second pose information. It should be understood that the position of the image can be represented by the set of coordinates of the four corner points of the virtual image in the preset spatial coordinate system.

[0062] Step S103: Based on the starting point coordinates, the emission direction, and the image position, determine the intersection of the virtual ray and the virtual image.

[0063] Based on the starting point coordinates, emission direction, and screen position of the virtual ray, the intersection of the virtual ray and the virtual screen is determined.

[0064] Step S20: In the case where the intersection is that the virtual ray and the virtual screen have an intersection point, determine the intersection point coordinates, and send the interaction event and the intersection point coordinates to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event.

[0065] When a virtual ray intersects with a virtual screen, the coordinates of the intersection point in a preset spatial coordinate system are determined, and the interactive event and the intersection point coordinates are sent to the head-mounted display device in the form of a data stream, so that the head-mounted display device can perform corresponding control operations based on the intersection point content and the interactive event (hereinafter referred to as the first control operation).

[0066] For example, an interaction event is user-initiated action information, which can be a physical operation (such as pressing a button) or a virtual operation (such as a specific gesture). A mapping relationship is pre-established between interaction events, intersection content, and control operations. For instance, if interaction event a is "clicking the confirmation button," intersection content a is "the video currently playing," and control operation a is "pausing playback," establishing a relationship between these three, then after receiving interaction event a and intersection content a, the head-mounted display device will find the corresponding control operation and execute the "pause playback" control operation. Similarly, if interaction event b is "clicking the confirmation button," intersection content b is "settings function," and control operation b is "confirming the selection," establishing a relationship between these three, then after receiving interaction event b and intersection content b, the head-mounted display device will find the corresponding control operation and execute the "confirm selection" control operation.

[0067] Step S30: In the case where the virtual ray and the virtual screen do not intersect, the interaction event is sent to the head-mounted display device so that the head-mounted display device can perform a second control operation based on the interaction event.

[0068] When there is no intersection between the virtual ray and the virtual screen, the interaction event is directly sent to the head-mounted display device so that the head-mounted display device can perform the corresponding control operation based on the interaction event (hereinafter referred to as the second control operation for distinction).

[0069] For example, a mapping relationship between interactive events and control operations can be established in advance. For instance, if interactive event c is "click the main menu button" and control operation a is "return to the main menu," establishing this association allows the head-mounted display device to find the corresponding control operation and execute the "return to the main menu" control operation after receiving a single interactive event c. Similarly, if interactive event d is "click the confirmation button" and control operation d is "confirm," establishing this association allows the head-mounted display device to find the corresponding control operation and execute the "confirm" control operation after receiving a single interactive event d.

[0070] In addition, it should be noted that the interactive data output by the communication module is interactive events, or the coordinates of the intersection of interactive events and the intersection point, which are represented as quaternions, and the format of the interactive data output is the OpenXR industry standard data format.

[0071] In one feasible implementation, the interaction event includes a gesture interaction event. Before step S10, the head-mounted display device interaction method of this application further includes:

[0072] Step A10: Monitor the pose change data of the target pen through the sensor module;

[0073] Step A20: Match the pose change data with the preset gesture data, and trigger the gesture interaction event when the pose change data and the preset gesture data are successfully matched.

[0074] It should be noted that a mapping relationship between common gestures and control operations is pre-set. For example, common gestures include "pointing at the virtual screen with a pen and tapping it once," "pointing at the virtual screen with a pen and tapping it twice," "waving the pen," and "tilting the pen forward / backward / left / right," etc., and each gesture is associated with a corresponding control operation, such as "confirm" or "return." The sensor data corresponding to each common gesture is also determined, i.e., the aforementioned preset gesture data.

[0075] The sensor module monitors the pose change data of the target pen, and the MCU compares the pose change data monitored by the sensor module with the preset gesture data. When it is determined that the pose change data and the preset gesture data are successfully matched, the gesture interaction event corresponding to the preset gesture data is triggered. The gesture interaction event represents the gesture information.

[0076] In another feasible embodiment, the interaction module further includes buttons, and the interaction events include button interaction events. Before step S10, the head-mounted display device interaction method of this application further includes:

[0077] Step B10: When a preset operation command triggered by the button is detected, the button interaction event is triggered.

[0078] It should be noted that a mapping relationship between button operations and preset operation commands is established in advance. For example, when the button operation is "single click", the corresponding preset operation command is "confirm", and when the button operation is "double click", the corresponding preset operation command is "return".

[0079] Users trigger preset operation commands by activating physical buttons on the interaction module. These preset operation commands, in turn, trigger button interaction events. It should be understood that each button interaction event represents a corresponding button operation.

[0080] Thus, this application provides two interaction methods: button interaction and gesture interaction.

[0081] This application embodiment designs an interactive module that communicates with a head-mounted display device. This interactive module can be installed on a pen used by the user in a teaching scenario. Specifically, the interactive module responds to interactive events by determining the intersection of a virtual ray emitted from the pen tip with the virtual image displayed on the head-mounted display device. If the intersection is found, the interactive event and intersection coordinates are sent to the head-mounted display device, allowing the head-mounted display device to perform a first control operation based on the intersection and the interactive event. If not, the interactive event is directly sent to the head-mounted display device, allowing the head-mounted display device to perform a second control operation based on the interactive event. In other words, the interactive module enables both directional and non-directional interaction with the head-mounted display device. This means that the target pen equipped with this interactive module simultaneously possesses interactive and writing functions, thereby avoiding the need for users to frequently switch between the pen and the interactive device in teaching scenarios. This improves the efficiency of user interaction with the head-mounted display device in teaching scenarios, ensures the continuity of user learning, and ultimately enhances the user's learning effect based on the head-mounted display device.

[0082] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. In addition, the interaction module includes a sensor module, and before step S10, it further includes:

[0083] Step C10: Using a preset reference surface as the target reference surface, the sensor module monitors the posture of the target pen to determine the angle between the pen tip pointing direction and the target reference surface, wherein the angle is an acute angle;

[0084] It should be noted that the initial reference plane is set in advance as the horizontal plane where the desktop is located in the current teaching scenario, that is, the preset reference plane mentioned above.

[0085] Using a preset reference surface as the target reference surface, the posture of the target pen is monitored by the sensor module to determine the angle between the pen tip pointing direction and the target reference surface, and the acute angle of the two angles formed by the pen tip pointing direction and the target reference surface is taken as the aforementioned angle.

[0086] Step C20: When the included angle is less than a preset angle, the ray interaction mode is activated to emit a virtual ray from the tip of the target pen.

[0087] It should be noted that the embodiments of this application do not limit the specific size of the above-mentioned preset angle. In this embodiment, the preset angle is 30 degrees. It should be understood that when the angle between the pen tip and the desktop is greater than or equal to 30 degrees, it is considered that the user is writing and there is no need to interact with the head-mounted display device.

[0088] When the included angle is detected to be less than the preset angle, a ray interaction event is automatically triggered. In response to the automatically triggered ray interaction event, the ray interaction mode is turned on to emit a virtual ray from the tip of the target pen, which means the enlightenment function and the interaction function are turned on.

[0089] Thus, this embodiment of the application detects the angle between the direction of the pen tip of the target pen being used by the user and the desktop, and determines whether the user is writing or needs to interact with the head-mounted display device. Specifically, it determines that the user has stopped writing, automatically activates the ray interaction module, and deactivates the ray interaction mode when it is confirmed that the user is writing, so as to avoid interfering with the user.

[0090] In this embodiment, the head-mounted display device interaction method of this application further includes:

[0091] Step D10: In response to a user-triggered calibration event, determine the plane where the target pen is located as the calibration plane, use the calibration plane as the target reference plane, and return to execute the step of monitoring the posture of the target pen through the sensor module and determining the angle between the pen tip pointing direction and the target reference plane.

[0092] It should be noted that, to avoid the impact of perceived offset of the desktop position during use on the angle monitoring, the interaction module provides a calibration function. Specifically, such as... Figure 3 The diagram illustrates the calibration process. After entering the calibration settings, the user sits upright at the table, places the target pen flat on the table perpendicular to the edge of the table on the user's side, with the pen tip away from the user, and maintains this position to reset the positioning and complete the calibration. It should be understood that resetting the target pen's position means reversing it, placing it horizontally on the teaching desktop and perpendicular to the user.

[0093] In response to a user-triggered calibration event, the sensor module monitors the pose of the target pen, determines the plane where the target pen is located as the calibration plane, and uses this calibration plane as the target reference plane. Then, it returns to the step of detecting the angle between the direction the target pen tip is pointing and the target reference plane, so that when the angle is detected to be less than a preset angle, the ray interaction mode is automatically activated.

[0094] Thus, by providing a desktop position calibration function, this application embodiment avoids the impact of the perceived offset of the desktop position during the use of the interaction module on the angle monitoring. By continuously monitoring the angle between the pen tip direction of the target pen used by the user and the desktop, it determines whether the user is writing or needs to interact with the head-mounted display device, so as to avoid interfering with the user when writing.

[0095] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and / or second embodiments described above can be referred to the above description and will not be repeated hereafter. In addition, the head-mounted display device interaction method of this application further includes:

[0096] Step E10: The sensor module of the interaction module monitors the motion state of the target pen. If the duration of the target pen being in a stationary state or in an abnormal motion state exceeds a preset duration, a low-power sleep mode is activated until a wake-up command is received.

[0097] Specifically, when the target pen is in the abnormal motion state, the angle change rate of the target pen is greater than a first preset threshold and / or the acceleration of the target pen is greater than a second preset threshold.

[0098] It should be noted that an upper limit for the rate of change of angle is preset, namely the first preset threshold, and an upper limit for acceleration is preset, namely the second preset threshold. If the rate of change of the angular velocity of the target pen is greater than the first preset threshold and / or the acceleration of the target pen is greater than the second preset threshold, the target pen is considered to be in an abnormal state such as pen spinning.

[0099] The sensor module of the interaction module monitors the movement state of the target pen. When the target pen is detected to be in a stationary state or an abnormal movement state, based on the duration of the stationary or abnormal movement state, and if the duration exceeds the preset duration, a low-power sleep mode is activated until a wake-up command is received from the user based on a button or head-mounted display device.

[0100] Thus, in this embodiment of the application, when the target pen is detected to be stationary for a long time (no change in angle or acceleration) or in an abnormal state such as pen spinning (drastic changes in angle or acceleration exceeding the normal range of use), a low-power sleep mode is activated until it is woken up, thereby achieving the effect of energy saving.

[0101] This application also provides an interactive device for a head-mounted display device. Please refer to... Figure 4 The head-mounted display device interaction device is applied to the interaction module, the interaction module is mounted on the target pen, and the head-mounted display device interaction device includes:

[0102] The response module 10 is used to respond to an interactive event and determine the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device.

[0103] The directional interaction module 20 is used to determine the intersection coordinates of the intersection point when the virtual ray and the virtual screen intersect, and send the interaction event and the intersection coordinates to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event.

[0104] The non-directional interaction module 30 is used to send the interaction event to the head-mounted display device when the intersection situation is that the virtual ray and the virtual screen do not intersect, so that the head-mounted display device can perform a second control operation based on the interaction event.

[0105] Optionally, the response module 10 is further configured to:

[0106] The first pose information of the target pen is determined, and the starting point and emission direction of the virtual ray emitted from the tip of the target pen are determined based on the first pose information. The starting point coordinates of the starting point in the preset spatial coordinate system are also determined.

[0107] The second pose information of the head-mounted display device is obtained, and the position of the virtual image displayed by the head-mounted display device in the preset spatial coordinate system is determined based on the second pose information.

[0108] Based on the starting point coordinates, the emission direction, and the image position, the intersection of the virtual ray and the virtual image is determined.

[0109] Optionally, the interaction module includes a sensor module, and the head-mounted display device interaction device of this application further includes a monitoring module, which is used for:

[0110] Using a preset reference surface as the target reference surface, the sensor module monitors the posture of the target pen to determine the angle between the pen tip pointing direction and the target reference surface, wherein the angle is an acute angle.

[0111] When the included angle is less than a preset angle, the ray interaction mode is activated to emit a virtual ray from the tip of the target pen.

[0112] Optionally, the head-mounted display device interaction device of this application further includes a calibration module, the calibration module being used for:

[0113] In response to a user-triggered calibration event, the plane where the target pen is located is determined as the calibration plane, the calibration plane is used as the target reference plane, and the process returns to the step of monitoring the posture of the target pen through the sensor module and determining the angle between the pen tip pointing direction and the target reference plane.

[0114] Optionally, the interaction events include gesture interaction events. The head-mounted display device interaction device of this application further includes a gesture interaction module, which is used for:

[0115] The sensor module monitors the pose change data of the target pen.

[0116] The pose change data is matched with preset gesture data, and the gesture interaction event is triggered when the pose change data and the preset gesture data are successfully matched.

[0117] Optionally, the interaction module further includes buttons, and the interaction events further include button interaction events. The head-mounted display device interaction apparatus of this application also includes a button interaction module, which is used for:

[0118] When a preset operation command triggered by the button is detected, the button interaction event is triggered.

[0119] Optionally, the interactive device for the head-mounted display of this application further includes a sleep module, which is used for:

[0120] The sensor module of the interaction module monitors the motion state of the target pen. If the duration of the target pen being in a stationary state or in an abnormal motion state exceeds a preset duration, a low-power sleep mode is activated until a wake-up command is received.

[0121] Specifically, when the target pen is in the abnormal motion state, the angle change rate of the target pen is greater than a first preset threshold and / or the acceleration of the target pen is greater than a second preset threshold.

[0122] The head-mounted display device interaction device provided in this application embodiment adopts the head-mounted display device interaction method in the above embodiments, which can solve the technical problem of how to improve the interaction efficiency with head-mounted display devices in teaching scenarios. Compared with the prior art, the beneficial effects of the head-mounted display device interaction device provided in this application embodiment are the same as the beneficial effects of the head-mounted display device interaction method provided in the above embodiments, and other technical features in the head-mounted display device interaction device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0123] This application provides an interaction module, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the head-mounted display device interaction method in Embodiment 1 above.

[0124] The following is for reference. Figure 5 It shows a structural schematic diagram suitable for implementing the interactive module of the embodiments of this application. Figure 5 The interactive module shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 5 As shown, the interactive module may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the interactive module. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the interaction module to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show interaction modules with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0126] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0127] The interactive module provided in this application, employing the head-mounted display device interaction method described in the above embodiments, can solve the technical problem of how to improve the efficiency of interaction with head-mounted display devices in teaching scenarios. Compared with the prior art, the beneficial effects of the interactive module provided in this application are the same as those of the head-mounted display device interaction method provided in the above embodiments, and other technical features in this interactive module are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0128] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0129] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0130] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the head-mounted display device interaction method in the above embodiments.

[0131] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0132] The aforementioned computer-readable storage medium may be included in the interactive module; or it may exist independently and not be assembled into the interactive module.

[0133] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the interaction module, the interaction module: in response to an interaction event, determines the intersection of a virtual ray emitted from the tip of the target pen with a virtual image displayed on the head-mounted display device; if the intersection is such that the virtual ray and the virtual image have an intersection point, determines the coordinates of the intersection point, and sends the interaction event and the intersection point coordinates to the head-mounted display device, so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event; if the intersection is such that the virtual ray and the virtual image do not have an intersection point, the interaction event is sent to the head-mounted display device, so that the head-mounted display device can perform a second control operation based on the interaction event.

[0134] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0137] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described head-mounted display device interaction method, thereby solving the technical problem of how to improve the efficiency of interaction with head-mounted display devices in teaching scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the head-mounted display device interaction method provided in the above embodiments, and will not be repeated here.

[0138] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the head-mounted display device interaction method described above.

[0139] The computer program product provided in this application can improve the efficiency of interaction with head-mounted display devices in teaching scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the head-mounted display device interaction method provided in the above embodiments, and will not be repeated here.

[0140] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for interacting with a head-mounted display device, characterized in that, The head-mounted display device interaction method is applied to an interaction module, the interaction module being mounted on a target pen, and the head-mounted display device interaction method includes: In response to an interactive event, determine the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device; In the case where the virtual ray and the virtual screen intersect, the coordinates of the intersection point are determined, and the interaction event and the intersection point coordinates are sent to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event. If the virtual ray and the virtual screen do not intersect, the interaction event is sent to the head-mounted display device so that the head-mounted display device can perform a second control operation based on the interaction event.

2. The method as described in claim 1, characterized in that, The step of determining the intersection of the virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device includes: The first pose information of the target pen is determined, and the starting point and emission direction of the virtual ray emitted from the tip of the target pen are determined based on the first pose information. The starting point coordinates of the starting point in the preset spatial coordinate system are also determined. The second pose information of the head-mounted display device is obtained, and the position of the virtual image displayed by the head-mounted display device in the preset spatial coordinate system is determined based on the second pose information. Based on the starting point coordinates, the emission direction, and the image position, the intersection of the virtual ray and the virtual image is determined.

3. The method as described in claim 1, characterized in that, The interaction module includes a sensor module. Before the step of determining the intersection of a virtual ray emitted from the tip of the target pen and a virtual image displayed on the head-mounted display device in response to an interaction event, the method further includes: Using a preset reference surface as the target reference surface, the sensor module monitors the posture of the target pen to determine the angle between the pen tip pointing direction and the target reference surface, wherein the angle is an acute angle. When the included angle is less than a preset angle, the ray interaction mode is activated to emit a virtual ray from the tip of the target pen.

4. The method as described in claim 3, characterized in that, The method further includes: In response to a user-triggered calibration event, the plane where the target pen is located is determined as the calibration plane, the calibration plane is used as the target reference plane, and the process returns to the step of monitoring the posture of the target pen through the sensor module and determining the angle between the pen tip pointing direction and the target reference plane.

5. The method as described in claim 3, characterized in that, The interaction events include gesture interaction events. Before the step of determining the preset interaction type corresponding to the interaction event in response to the interaction event, the method further includes: The sensor module monitors the pose change data of the target pen. The pose change data is matched with preset gesture data, and the gesture interaction event is triggered when the pose change data and the preset gesture data are successfully matched.

6. The method as described in claim 1, characterized in that, The interaction module further includes buttons, and the interaction events further include button interaction events. Before the step of determining the preset interaction type corresponding to the interaction event in response to the interaction event, the method further includes: When a preset operation command triggered by the button is detected, the button interaction event is triggered.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The sensor module of the interaction module monitors the motion state of the target pen. If the duration of the target pen being in a stationary state or in an abnormal motion state exceeds a preset duration, a low-power sleep mode is activated until a wake-up command is received. Specifically, when the target pen is in the abnormal motion state, the angle change rate of the target pen is greater than a first preset threshold and / or the acceleration of the target pen is greater than a second preset threshold.

8. An interactive device for a head-mounted display device, characterized in that, The head-mounted display device interaction device is applied to the interaction module, the interaction module is mounted on the target pen, and the head-mounted display device interaction device includes: A response module is used to respond to interactive events and determine the intersection of a virtual ray emitted from the tip of the target pen with the virtual image displayed on the head-mounted display device. A directional interaction module is used to determine the intersection coordinates of the virtual ray and the virtual screen when the intersection situation is that the virtual ray and the virtual screen have an intersection point, and send the interaction event and the intersection coordinates to the head-mounted display device so that the head-mounted display device can perform a first control operation based on the intersection point and the interaction event; A non-directional interaction module is used to send the interaction event to the head-mounted display device when the intersection situation is that the virtual ray and the virtual screen do not intersect, so that the head-mounted display device can perform a second control operation based on the interaction event.

9. An interactive module, characterized in that, The interaction module includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the head-mounted display device interaction method as described in any one of claims 1 to 7.

10. An interactive pen, characterized in that, The interactive pen includes the interactive module as described in claim 9.

Images