A method for operating a three-dimensional interface, a head-mounted device, and an operating system

Abstract

The application relates to the technical field of human-computer interaction, and provides a three-dimensional interface operation method, a head control device and an operation system, the head control device is connected with a master control ring through Bluetooth, the master control ring is connected with an auxiliary control ring through Bluetooth, the two rings can be worn on two fingers of one hand or placed on a desktop, so that two modes of gesture interaction and desktop touch interaction are realized, and the use scene is more extensive. In the gesture interaction mode, a sighting line is determined according to self position data and master control ring position data, and after the sighting line aims at a target, finger pinching operation is determined according to the distance between the two fingers determined according to the position data of the other party collected by the two rings; in the desktop touch interaction mode, a sighting line is determined according to the position data of a fingertip collected by any ring, and after the sighting line aims at a target, finger desktop knocking operation is determined according to the rotation change collected by the corresponding ring. The two interaction modes are simple to switch, and can meet different use scene requirements.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and provides a method for operating a three-dimensional interface, a head control device, and an operating system. Background Technology

[0002] Head-mounted displays (HMDs), as a new type of display, place great importance on the user experience through their interactive design. Currently, common interaction methods include:

[0003] (1) Touch or button: Some HMDs are designed with touchpads or physical buttons, which users can interact with by touching or pressing, making them easy to use. However, due to the limited operating space on HMDs, they are generally used for simple interactive operations.

[0004] (2) Eye tracking: Using eye tracking technology, users can interact with HMD by moving their eyes and gazing. It is convenient to operate, but the accuracy of the interaction is poor and it has high requirements for the use scenario.

[0005] (3) Voice commands: Through voice recognition technology, users can directly issue commands to the HMD, which is convenient to operate, but the accuracy of interaction is poor and the requirements for the use scenario are high.

[0006] (4) Handheld controller, similar to a handle, can provide more precise and complex interactive operations, but it increases the user's equipment burden and is not convenient for the user to carry;

[0007] (5) Gesture recognition: Users can interact with HMD through different gestures (such as waving, pinching, stretching, etc.), which is convenient to operate. However, users need to raise both hands continuously to operate. After long-term use, users will consume a lot of physical strength and experience arm fatigue, which is not conducive to long-term use.

[0008] Therefore, providing a lightweight and accurate interaction method is of great research value for HMD. Summary of the Invention

[0009] This application provides a three-dimensional interface operation method, head control device, and operating system to improve the accuracy of interaction in head-mounted display devices and reduce the user's interaction burden.

[0010] In a first aspect, embodiments of this application provide a method for operating a three-dimensional interface, applied to a head-controlled device, wherein the head-controlled device is connected to a main control ring via Bluetooth, and the main control ring is connected to an auxiliary control ring via Bluetooth, the method comprising:

[0011] The interaction mode is determined based on the indication information sent by the main control ring; wherein, the indication information is triggered based on user operation.

[0012] When the interaction mode is gesture interaction mode, the aiming line in the three-dimensional interface is determined based on its own position data and the pose data of the main control ring. After the aiming line is aimed at the target in the three-dimensional interface, the distance between the two fingers is determined based on the position data of the other party collected by the main control ring and the auxiliary control ring respectively, so as to realize the confirmation operation triggered by finger pinching. In the gesture interaction mode, the main control ring and the auxiliary control ring are worn on two fingers of one hand.

[0013] When the interaction mode is a desktop touch interaction mode, the aiming line in the three-dimensional interface is determined based on the fingertip position data collected by any ring. After the aiming line is aimed at the target in the three-dimensional interface, the rotation change is determined based on the rotation data collected by the corresponding ring to realize the confirmation operation triggered by tapping the desktop. In the desktop touch interaction mode, the main control ring and the auxiliary control ring are placed on the desktop.

[0014] The effective effects of the above technical solution are as follows: the main control ring and the auxiliary control ring have two different usage methods. One is to wear them on two fingers of one hand to achieve a gesture interaction mode, and the other is to place them on a table to achieve a desktop touch interaction mode. In this way, during interaction, the user can trigger instruction information by operating the main control ring according to the actual interaction needs. The head control device determines the interaction mode according to the instruction information, realizing flexible selection of interaction modes in different scenarios, making the operation of the head-mounted display device more flexible and diverse, and its application scenarios more extensive. In gesture interaction mode, the head-controlled device determines the aiming line in the 3D interface based on its own position data and the pose data of the main control ring. Since the aiming line is determined using data from both the head and hand, the aiming process in the 3D interface is more stable and accurate, thus improving the interaction precision. Furthermore, after the aiming line is aimed at the target in the 3D interface by hand movement, the distance between the two fingers is determined based on the position data of the target collected by the main and auxiliary control rings to determine whether the user has confirmed the target, thus realizing gesture interaction. In desktop touch interaction mode, the head-controlled device determines the aiming line in the 3D interface based on the fingertip position data collected by the ring. After the aiming line is aimed at the target by sliding the finger on the desktop, the amount of rotation change is determined based on the rotation data collected by the corresponding ring to determine whether the user has confirmed the target, thus realizing desktop touch interaction.

[0015] Optionally, the pose data includes position data and rotation data, and determining the aiming line in the three-dimensional interface based on its own position data and the pose data of the main control ring includes:

[0016] The starting point of the aiming line is determined based on its own position data and the position data of the main control ring.

[0017] Using the starting point as the origin, a ray is rendered towards the three-dimensional interface according to the direction represented by the rotation data of the main control ring to obtain the aiming line.

[0018] The beneficial effects of the above technical solution are as follows: In the gesture interaction mode, the head control device determines the starting point and direction of the aiming line based on its own position data, the position data of the main control ring, and the rotation data. This enables the joint calibration of the aiming line in the three-dimensional interface using both the head and the hand, improving the stability and accuracy of the aiming line. At the same time, it allows users to select targets by moving the aiming line in the three-dimensional interface through the movement of their fingers, making the operation simple and convenient.

[0019] Optionally, when the main control ring is worn on the left hand, the starting point position is calculated as follows:

[0020] x0=x h -x t *a

[0021] y0=y h -y t *b

[0022] z0=z h

[0023] When the main control ring is worn on the right hand, the starting point position is calculated as follows:

[0024] x0=x h +x t *a

[0025] y0=y h -y t *b

[0026] z0=z h

[0027] Among them, (x h y h , z h (x) represents the position coordinates of the head control device. t y t , z t ) represents the position coordinates of the main control ring, a is the offset coefficient of the X-axis, and b is the offset coefficient of the Y-axis.

[0028] The beneficial effects of the above technical solution are as follows: Two calculation methods for the trigger point position are designed for the two cases where the main control ring and the auxiliary control ring are worn on the right hand and the left hand. Through the above two calculation methods, the starting point of the aiming line is located in the middle of the hand and the head. Since the head is generally kept still when aiming, the stability of the starting point position calculation is improved, resulting in higher aiming accuracy and stability.

[0029] Optionally, the step of determining the distance between two fingers based on the position data of the other party collected by the main control ring and the auxiliary control ring respectively to realize the confirmation operation triggered by finger pinching includes:

[0030] The first position coordinates of the main control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the main control ring is located; and the second position coordinates of the auxiliary control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the auxiliary control ring is located.

[0031] Based on the first position coordinates and the second position coordinates, the distance between the two fingers is determined, and based on the comparison result of the distance between the two fingers and a preset distance threshold, it is determined whether a confirmation operation triggered by finger pinching has occurred.

[0032] The beneficial effects of the above technical solution are as follows: Since the main control ring and the auxiliary control ring are worn on two fingers of one hand, the distance between the two fingers can be determined by their position coordinates. Based on a comparison of this distance with a preset distance threshold, it can be determined whether the user has performed a pinch gesture, thus enabling gesture interaction on the head-mounted display device. Furthermore, by calculating the ring's position using the fingertip coordinates and at least one knuckle coordinate, the accuracy of the two-finger distance calculation is improved, thereby enhancing the determination of the pinch gesture and ultimately increasing the interaction precision.

[0033] Optionally, when the main control ring is worn on the thumb and the auxiliary control ring is worn on a finger other than the thumb, the first position coordinates are calculated as follows:

[0034] x t =(x tt *c1+x tj *c2) / (c1+c2)

[0035] y t =(y tt *c1+y tj *c2) / (c1+c2)

[0036] z t =(z tt *c1+z tj *c2) / (c1+c2)

[0037] The second position coordinate is calculated as follows:

[0038] x f =(x ft *d1+x fj1 *d2+x fj2 *d3) / (d1+d2+d3)

[0039] y f =(y ft *d1+y fj1 *d2+y fj2 *d3) / (d1+d2+d3)

[0040] z f =(z ft *d1+z fj1 *d2+z fj2 *d3) / (d1+d2+d3)

[0041] Among them, (x tt y tt , z tt (x) represents the coordinates of the fingertip position of the thumb. tj y tj , z tj (x) represents the coordinates of the middle joint of the thumb; ft y ft , z ft (x) represents the coordinates of the fingertip positions of the other fingers. fj1 y fj1 , z fj1 (x) represents the coordinates of the first joint position of the other fingers. fj2 y fj2 , z fj2 ) represents the coordinates of the second joint of the other fingers, and c1, c2, d1, d2 and d3 are weighted average parameters.

[0042] The beneficial effects of the above technical solution are as follows: by wearing the main control ring on the thumb and the auxiliary control ring on other fingers, the distance between the two fingers can be increased, thereby avoiding misjudgment of gestures when the distance between the two fingers is small and improving the accuracy of interaction.

[0043] Optionally, determining the aiming line in the three-dimensional interface based on the fingertip position data collected by any ring includes:

[0044] Based on the fingertip position data collected by any ring, a touch calibration point on the Overlay calibration plane is determined; the Overlay calibration plane is a transparent screen, located on the upper layer of the three-dimensional interface and parallel to the three-dimensional interface, and the origin of the coordinates of the Overlay calibration plane is the position of any ring;

[0045] Using the touch calibration point as the origin, a ray perpendicular to the three-dimensional interface is rendered to obtain the aiming line.

[0046] The beneficial effects of the above technical solution are as follows: In the desktop touch interaction mode, the head control device determines the starting position of the aiming line based on the aiming method of the Overlay calibration plane according to the position data of the fingertip corresponding to any ring, thereby completing the rendering of the aiming line in the three-dimensional interface. The aiming accuracy is high. In addition, in this mode, the user can change the aiming line in the three-dimensional interface by moving the fingertip on the desktop, which is convenient to operate.

[0047] Optionally, the aspect ratio of the touch operation area of ​​the ring placed on the desktop is the same as the aspect ratio of the Overlay calibration plane. The step of determining the touch calibration point on the Overlay calibration plane based on the fingertip position data collected by any ring includes:

[0048] Based on the distance from the fingertip to the corresponding ring within the touch operation area of ​​any ring and the aspect ratio, determine the distance between the mapping point on the Overlay calibration plane and the origin of the coordinate system.

[0049] The orientation of the mapping point on the Overlay calibration plane is determined based on the orientation of the fingertip relative to the ring within the touch operation area of ​​any ring;

[0050] Based on the distance and direction corresponding to the mapping point, the touch calibration point on the Overlay calibration plane is determined.

[0051] The beneficial effects of the above technical solution are as follows: by creating an Overlay calibration plane, the starting position of the aiming line is switched to the touch calibration point on the plane. Since the length and width ratio of the set touch operation area of ​​the ring is the same as the length and width ratio of the Overlay calibration plane, the movement of the fingertip in the touch operation area of ​​the ring on the desktop can drive the movement of the aiming line in the three-dimensional interface, thereby realizing desktop touch operation.

[0052] Optionally, determining the amount of rotation change based on the rotation data collected by the corresponding ring to achieve the confirmation operation triggered by tapping the table includes:

[0053] The amount of rotational change is determined based on the acceleration collected by the corresponding ring; wherein, the change in acceleration is generated when the corresponding ring is subjected to pressure, and the pressure is generated by the vibration of the table caused by the fingertip tapping the table.

[0054] Based on the comparison between the rotation change and the preset rotation threshold, it is determined whether a confirmation operation triggered by tapping the desktop has occurred.

[0055] The beneficial effect of the above technical solution is that it detects whether the fingertip taps the desktop by changing the amount of rotation, thereby realizing the confirmation operation in the desktop touch interaction mode.

[0056] Secondly, embodiments of this application provide a head control device, including a posture sensor, a Bluetooth module, a storage chip, and a computing chip;

[0057] The Bluetooth module is used to connect with the main control ring via Bluetooth, and the main control ring is connected with the auxiliary control ring via Bluetooth.

[0058] The storage chip includes a data storage area and a program storage area. The data storage area is used to store the data collected by itself and the data collected by the ring. The program storage area is used to store the computer program.

[0059] The computing chip performs the following operations according to the computer program:

[0060] The Bluetooth module receives indication information sent by the master ring and determines the interaction mode based on the indication information; wherein, the indication information is triggered based on user operation.

[0061] When the interaction mode is gesture interaction mode, the aiming line in the three-dimensional interface is determined based on the position data of the pose sensor and the pose data of the main control ring. After the aiming line is aimed at the target in the three-dimensional interface, the Bluetooth module receives the position data of the other party sent by the main control ring and the auxiliary control ring, and determines the distance between the two fingers based on the other party's position data to realize the confirmation operation triggered by finger pinching. In the gesture interaction mode, the main control ring and the auxiliary control ring are worn on two fingers of one hand.

[0062] When the interaction mode is desktop touch interaction mode, the Bluetooth module receives fingertip position data collected by any ring, determines the aiming line in the three-dimensional interface based on the fingertip position data, and after the aiming line aims at the target in the three-dimensional interface, the Bluetooth module receives rotation data collected by the corresponding ring, and determines the amount of rotation change based on the rotation data to realize the confirmation operation triggered by tapping the desktop; wherein, in the desktop touch interaction mode, the main control ring and the auxiliary control ring are placed on the desktop.

[0063] Optionally, the pose data includes position data and rotation data. The step of determining the aiming line in the 3D interface based on the position data collected by the pose sensor and the pose data of the main control ring is as follows:

[0064] The starting point of the aiming line is determined based on the position data of the pose sensor and the position data of the main control ring.

[0065] Using the starting point as the origin, a ray is rendered towards the three-dimensional interface according to the direction represented by the rotation data of the main control ring to obtain the aiming line.

[0066] Optionally, when the main control ring is worn on the left hand, the starting point position is calculated as follows:

[0067] x0=x h -x t *a

[0068] y0=y h -y t *b

[0069] z0=z h

[0070] When the main control ring is worn on the right hand, the starting point position is calculated as follows:

[0071] x0=x h +x t *a

[0072] y0=y h -y t *b

[0073] z0=z h

[0074] Among them, (x h y h , z h (x) represents the position coordinates of the head control device. t y t , z t ) represents the position coordinates of the main control ring, a is the offset coefficient of the X-axis, and b is the offset coefficient of the Y-axis.

[0075] Optionally, the step of determining the distance between two fingers based on the other party's position data to achieve the confirmation operation triggered by finger pinching is as follows:

[0076] The first position coordinates of the main control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the main control ring is located; and the second position coordinates of the auxiliary control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the auxiliary control ring is located.

[0077] Based on the first position coordinates and the second position coordinates, the distance between the two fingers is determined, and based on the comparison result of the distance between the two fingers and a preset distance threshold, it is determined whether a confirmation operation triggered by finger pinching has occurred.

[0078] When the main control ring is worn on the thumb and the auxiliary control ring is worn on other fingers besides the thumb, the first position coordinates are calculated as follows:

[0079] x t =(x tt *c1+x tj *c2) / (c1+c2)

[0080] y t =(y tt *c1+y tj *c2) / (c1+c2)

[0081] z t =(z tt *c1+z tj *c2) / (c1+c2)

[0082] The second position coordinate is calculated as follows:

[0083] x f =(x ft *d1+x fj1 *d2+x fj2 *d3) / (d1+d2+d3)

[0084] y f =(y ft *d1+y fj1 *d2+y fj2 *d3) / (d1+d2+d3)

[0085] z f =(z ft *d1+z fj1 *d2+z fj2 *d3) / (d1+d2+d3)

[0086] Among them, (x tt y tt , z tt (x) represents the coordinates of the fingertip position of the thumb. tj y tj , z tj (x) represents the coordinates of the middle joint of the thumb; ft y ft , z ft (x) represents the coordinates of the fingertip positions of the other fingers. fj1 y fj1 , z fj1 (x) represents the coordinates of the first joint position of the other fingers. fj2 y fj2 , z fj2) represents the coordinates of the second joint of the other fingers, and c1, c2, d1, d2 and d3 are weighted average parameters.

[0087] Optionally, the step of determining the aiming line in the three-dimensional interface based on the fingertip position data specifically involves:

[0088] Based on the fingertip position data collected by any ring, a touch calibration point on the Overlay calibration plane is determined; the Overlay calibration plane is a transparent screen, located on the upper layer of the three-dimensional interface and parallel to the three-dimensional interface, and the origin of the coordinates of the Overlay calibration plane is the position of any ring;

[0089] Using the touch calibration point as the origin, a ray perpendicular to the three-dimensional interface is rendered to obtain the aiming line.

[0090] Optionally, the aspect ratio of the touch operation area of ​​the ring placed on the desktop is the same as the aspect ratio of the Overlay calibration plane. The specific operation of determining the touch calibration point on the Overlay calibration plane based on the fingertip position data collected by any ring is as follows:

[0091] Based on the distance from the fingertip to the corresponding ring within the touch operation area of ​​any ring and the aspect ratio, determine the distance between the mapping point on the Overlay calibration plane and the origin of the coordinate system.

[0092] The orientation of the mapping point on the Overlay calibration plane is determined based on the orientation of the fingertip relative to the ring within the touch operation area of ​​any ring;

[0093] Based on the distance and direction corresponding to the mapping point, the touch calibration on the Overlay calibration plane is determined.

[0094] Optionally, the step of determining the rotation change based on the rotation data to implement the confirmation operation triggered by tapping the desktop is as follows:

[0095] The amount of rotational change is determined based on the acceleration collected by the corresponding ring; wherein, the change in acceleration is generated when the corresponding ring is subjected to pressure, and the pressure is generated by the vibration of the table caused by the fingertip tapping the table.

[0096] Based on the comparison between the rotation change and the preset rotation threshold, it is determined whether a confirmation operation triggered by tapping the desktop has occurred.

[0097] Thirdly, embodiments of this application provide a three-dimensional interface operating system, including a head control device, a main control ring, and an auxiliary control ring;

[0098] The main control ring and the auxiliary control ring each include a first posture sensor, a first Bluetooth module, a first storage chip, and a mode button, wherein:

[0099] The first pose sensor is used to collect rotation data and position data, and stores the data through the first storage chip;

[0100] The first Bluetooth module is used for Bluetooth connection between the rings and Bluetooth connection with the head control device;

[0101] The mode button is used to send an instruction message to the head control device to determine the interaction mode in response to user operation;

[0102] The head control device includes a second pose sensor, a second Bluetooth module, a second storage chip, and a second computing chip.

[0103] The second pose sensor is used to collect position and rotation data;

[0104] The second Bluetooth module is used to establish a Bluetooth connection with the main control ring;

[0105] The second storage chip includes a program storage area and a data storage area, wherein the data storage area is used to store data collected by itself and data collected by the ring;

[0106] The computing chip is used to execute the method described in any one of the first aspects according to the computer program stored in the program storage area.

[0107] Fourthly, embodiments of this application provide a head-mounted display device, the head-mounted display device including a head control device for implementing any of the three-dimensional interface operation methods provided in the first aspect.

[0108] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a head-controlled device to perform an operation method of any of the three-dimensional interfaces provided in the first aspect. Attached Figure Description

[0109] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0110] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0111] Figure 2A structural diagram of the operating system provided in the embodiments of this application;

[0112] Figure 3A A scene diagram of a gesture interaction mode provided in an embodiment of this application;

[0113] Figure 3B This is a scene diagram illustrating another gesture interaction mode provided in an embodiment of this application.

[0114] Figure 4 This is a scenario diagram of a desktop touch interaction mode provided in an embodiment of this application;

[0115] Figure 5 A flowchart illustrating the operation method of the head-mounted display device provided in the embodiments of this application;

[0116] Figure 6 A flowchart illustrating the determination of the aiming line under the gesture interaction mode provided in this application embodiment;

[0117] Figure 7 This is a schematic diagram illustrating the movement of the aiming line under the gesture interaction mode provided in the embodiments of this application;

[0118] Figure 8 This is a flowchart of the confirmation operation under the gesture interaction mode provided in the embodiments of this application;

[0119] Figure 9 This is a schematic diagram of gesture interaction provided in an embodiment of this application;

[0120] Figure 10 A flowchart illustrating the determination of the aiming line in a desktop touch interaction mode provided in this application embodiment;

[0121] Figure 11 A schematic diagram of the coordinate axes for the Overlay calibration plane is provided for embodiments of this application;

[0122] Figure 12 A flowchart illustrating the method for mapping touch calibration points in a desktop touch interaction mode provided in this application embodiment;

[0123] Figure 13 This is a flowchart of the confirmation operation in the desktop touch interaction mode provided in the embodiments of this application;

[0124] Figure 14 This is a schematic diagram of desktop touch interaction operation provided in an embodiment of this application;

[0125] Figure 15 This is a timing diagram illustrating the interaction between the operating system and the head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0126] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0127] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0128] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0129] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0130] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0131] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0132] The design concept of the embodiments of this application will be summarized below in conjunction with application scenarios.

[0133] Currently, most head-mounted display devices rely on the device itself for gesture functions, which increases the size and weight of the device. In addition, the gesture functions of the device itself are simple, usually with only one operating mode. Users need to raise both hands to operate during use, which can cause arm fatigue over time.

[0134] In view of this, this application provides a method for operating a three-dimensional interface, including a head control device, a main control ring, and an auxiliary control ring. The main control ring and the auxiliary control ring can be worn on two fingers of one hand or placed on a table, thereby realizing two interaction modes: gesture interaction and desktop touch interaction. In gesture interaction mode, the head control device determines the aiming line in the three-dimensional interface based on its own position data, as well as the position and rotation data of the main control ring, and determines the distance between the two fingers based on the position data of the main control ring and the auxiliary control ring to realize the confirmation operation triggered by finger pinching. Since the aiming line determination in gesture interaction mode uses data from both the head and hand, the aiming process of the aiming line in the three-dimensional interface is more stable and accurate, thereby improving the accuracy of finger pinching operation judgment and ensuring the accuracy of interaction. In desktop touch interaction mode, the head control device determines the aiming line in the three-dimensional interface based on the fingertip position data collected by the ring on either side, and determines the rotation change based on the rotation data of the corresponding ring to realize the confirmation operation triggered by tapping the table. By placing the ring on the table to form a desktop touch area, the user can interact by moving their fingertips and tapping the table. In this way, in practical applications, users can trigger instruction information by operating the main control ring according to their actual interaction needs. The head control device then determines the interaction mode based on the instruction information, thereby enabling flexible selection of interaction modes in different scenarios. This makes the operation of head-mounted display devices more flexible and diverse, and its application scenarios more extensive.

[0135] See Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application. It includes a head-mounted display device and an operating system for interaction. The operating system includes a head control device, a main control ring, and an auxiliary control ring. The head control device is fixed to one side of the head-mounted display device. The main control ring and the auxiliary control ring can be worn on the user's wrist for gesture interaction or placed on a table for desktop touch interaction.

[0136] Optionally, the head-mounted display devices in this application include, but are not limited to, AR glasses, VR glasses, and MR glasses.

[0137] It should be noted that, Figure 1 This is just one example of an application scenario; optionally, the head control device can also be integrated into the head-mounted display device.

[0138] See Figure 2This is a structural diagram of the operating system provided in this application embodiment. The main control ring includes a first posture sensor, a first Bluetooth module, a first storage chip, and a mode button. The first posture sensor is used to collect rotation and position data. The first Bluetooth module is used to connect with the head control device and the auxiliary control ring via Bluetooth. The first storage chip is used to store the data it collects and the data collected by the auxiliary control ring. The mode button is used to send an instruction to the head control device to confirm the interaction mode in response to user operation. The auxiliary control ring also includes a first posture sensor, a first Bluetooth module, and a first storage chip. The first posture sensor is used to collect rotation and position data. The first Bluetooth module is used to connect with the main control ring via Bluetooth. The first storage chip is used to store the data it collects. The head control device includes a second posture sensor, a second Bluetooth module, a second storage chip, and a computing chip. The second posture sensor is used to collect position and rotation data. The second Bluetooth module is used to connect with the main control ring via Bluetooth. The second storage chip includes a data storage area and a program storage area. The data storage area stores the data it collects and the data collected by the ring. The computing chip performs calculations on the data according to the computer program stored in the program storage area to achieve interactive operation.

[0139] Optionally, the first pose sensor includes a first IMU sensor and a first ToF sensor. In this case, the first IMU sensor in the main control ring is used to collect finger rotation data, and the first ToF sensor is used to collect position data of the auxiliary control ring. Similarly, the first IMU sensor in the auxiliary control ring is used to collect finger rotation data, and the first ToF sensor is used to collect position data of the main control ring. The second pose sensor includes a second IMU sensor and a second ToF sensor. In this case, the second IMU sensor is used to collect head rotation data, and the second ToF sensor is used to collect its own position data and the position data of the main control ring.

[0140] Optionally, the master and slave rings may also include adjustment knobs for securing the rings to the finger.

[0141] Optionally, the head control device may also include a retaining clip for securing the head control device to the head-mounted display device.

[0142] Optionally, the main control ring and the auxiliary control ring may also include a first system indicator light, and the head control device may also include a second system indicator light to indicate the Bluetooth connection status among the three.

[0143] Optionally, the main control ring and the auxiliary control ring may also include a first switch button, and the head control device may also include a second switch button for controlling the switching of the device.

[0144] Optionally, the mode button and the first switch button on the main control ring can be the same, and different button settings can be used to represent different functions.

[0145] In the embodiments of this application, the usage of the head control device, the main control ring, and the auxiliary control ring can be explained to the user in the instruction manual so that the user can use them correctly according to the instructions.

[0146] In one example, the master control ring and the slave control ring can be worn on one hand to enable gesture interaction with the head-mounted display device.

[0147] Taking the head-controlled device fixed to the outside of the head-mounted display device as an example, such as Figure 3A As shown, when the head control device is fixed to the left temple of the head-mounted display device by the second fixing buckle, the main control ring and the auxiliary control ring are worn on the left hand, as shown. Figure 3B As shown, when the head control device is fixed to the right temple of the head-mounted display device by the second fixing buckle, the main control ring and the auxiliary control ring are worn on the right hand.

[0148] It should be noted that when the head control device is fixed inside the head-mounted display device, it can be integrated on the left side of the device, or on the right side of the device, or it can be flexibly set according to the actual size of the device.

[0149] Generally, the primary control ring and the secondary control ring can be worn on two different fingers of the same hand.

[0150] In practical applications, the main control ring can be worn at the base of the thumb and fixed by adjusting the knob, while the auxiliary control ring can be worn at the base of other fingers on the same hand (such as the index, middle, ring, or little finger) and fixed by adjusting the knob, thereby increasing the gap between the two fingers and improving the gesture recognition rate.

[0151] In one example, the master control ring and the slave control ring can be placed on a flat tabletop, such as... Figure 4 As shown, the main control ring can be placed on the right side of the table, and the auxiliary control ring on the left side; alternatively, the main control ring can be placed on both sides of the table, and the auxiliary control ring on the right side. In practice, the rings can be placed vertically on the table, with the first system indicator light facing the direction the finger touches the table. This maintains the stability of the ring placement and allows the first ToF sensor inside the ring to be closer to the table and facing the hand, thereby improving the success rate of desktop touch recognition.

[0152] In the embodiments of this application, a Bluetooth connection must be established before interacting with the aforementioned operating system.

[0153] Taking a head-controlled device independent of the head-mounted display as an example, in specific implementation, after the rings are worn or placed, briefly press the power buttons on the head-controlled device, main control ring, and auxiliary control ring to activate them respectively. During activation, the system indicator lights on the head-controlled device, main control ring, and auxiliary control ring are blue, and turn off after successful activation to enter the Bluetooth pairing process. During pairing, press and hold the second power button on the head-controlled device. The second system indicator light will start flashing green, flashing once every second. After flashing five times, it indicates that the head-controlled device has successfully connected to the head-mounted display via Bluetooth. If the connection fails, the system indicator light on the head-controlled device will flash red. The user can re-pair by pressing and holding the power button again until the Bluetooth connection is successful. Furthermore, press and hold the first power button on the main control ring. The first system indicator light will start flashing green, flashing once every second. After flashing five times, it indicates that the main control ring has successfully connected to the head-controlled device via Bluetooth. If the connection fails, the first system indicator light on the main control ring will flash red. The user can re-pair by pressing and holding the first power button again until the Bluetooth connection is successful. Finally, press and hold the first power button on the auxiliary control ring. The first system indicator light on the auxiliary control ring will start flashing green, flashing once every second. After flashing 5 times, it indicates that the auxiliary control ring and the main control ring have successfully connected via Bluetooth. If the connection fails, the first system indicator light on the auxiliary control ring will flash red. The user can re-pair the ring by pressing and holding the first power button again until the Bluetooth connection is successful.

[0154] It should be noted that the embodiments of this application do not impose restrictive requirements on the indication method of the system indicator lights and the usage method of the switch buttons, which can be adjusted according to actual needs.

[0155] It should be noted that when the head control device is integrated into the head-mounted display device, the Bluetooth connection process between the head control device and the head-mounted display device can be skipped, and the process can directly proceed to the Bluetooth connection process between the main control ring and the head-mounted display device.

[0156] This application provides a method for operating a three-dimensional interface displayed on a head-mounted display device. This method is executed by a head-controlled device, such as... Figure 5 The steps shown are as follows:

[0157] S501: Determine the interaction mode based on the instruction information sent by the main control ring. When it is a gesture interaction mode, execute S502; when it is a desktop touch interaction mode, execute S504.

[0158] Since the main control ring and the auxiliary control ring have two usage scenarios—one is to wear them on two fingers of one hand, and the other is to place them on a table—there are different interaction methods in different scenarios. Therefore, both gesture interaction and table touch control interaction modes can be realized.

[0159] In one example, after a successful Bluetooth connection, the user wears the main control ring and the auxiliary control ring on two fingers of one hand. The head-controlled device defaults to gesture interaction mode. In this mode, the user does not need to operate the main control ring's power button; that is, the main control ring does not send any instructions to the head-controlled device. When the user wants to switch to desktop touch interaction mode, they simply remove the main control ring and the auxiliary control ring and place them on a table. A short press of the main control ring's power button sends an instruction to the head-controlled device, notifying it to enter desktop touch interaction mode.

[0160] In another example, when no default interaction mode is set, after a successful Bluetooth connection, the user can send instructions to select an interaction mode by operating the master ring.

[0161] For example, when a user wants to enter gesture interaction mode, the head control device is fixed to one side of the head-mounted display device, and the main control ring and auxiliary control ring are worn on two fingers of one hand. The first switch button of the main control ring is pressed twice in succession to send an indication message including two consecutive level signals to the head control device, thereby informing the head control device to enter gesture interaction mode. When a user wants to enter desktop touch interaction mode, the head control device is fixed to one side of the head-mounted display device, and the main control ring and auxiliary control ring are placed on the table. The switch button of the main control ring is pressed three times in succession to send an indication message including three consecutive level signals to the head-mounted display device, thereby informing the head control device to enter desktop touch interaction mode.

[0162] It should be noted that the embodiments of this application do not impose restrictive requirements on the pressing method (such as long press, short press, number of times) of the switch button of the main control ring and the function indicated by each pressing method.

[0163] S502: Determines the aiming line in the three-dimensional interface based on its own position data and the pose data of the master control ring.

[0164] In gesture interaction mode, the user operates the device using the hand wearing the main control ring and the auxiliary control ring, determining the aiming line in the 3D interface of the head-mounted display by moving and rotating the hand up, down, left, right, and right. See details... Figure 6 It mainly includes the following steps:

[0165] S5021: Determines the starting point of the aiming line based on its own position data and the position data of the main control ring.

[0166] When entering gesture interaction mode, the ToF sensor of the head-controlled device collects its own position data and the position data of the main control ring in real time, storing the collected position data in the data storage area of ​​the storage chip. Simultaneously, the IMU sensor of the main control ring also collects its own rotation data in real time, storing the collected rotation data in the data storage area of ​​the storage chip and transmitting it to the head-controlled device via Bluetooth. The head-controlled device also stores the rotation data received via Bluetooth in its storage chip. The data in the storage chip is then fed into the computing chip to calculate the starting position of the aiming line in the 3D interface.

[0167] When the master control ring is worn on the left hand, the starting point is located between the left side of the head (head control device) and the left hand (master control ring), calculated as follows:

[0168] x0=x h -x t *a Formula 1

[0169] y0=y h -y t *b Formula 2

[0170] z0=z h Formula 3

[0171] When the master control ring is worn on the right hand, the starting point is the midpoint between the right side of the head (head control device) and the right hand (master control ring), calculated as follows:

[0172] x0=x h +x t *a Formula 4

[0173] y0=y h -y t *b Formula 5

[0174] z0=z h Formula 6

[0175] Among them, (x h y h , z h (x) represents the position coordinates of the head control device. t y t , z t ) is the position coordinate of the main control ring, a is the offset coefficient of the X-axis, and b is the offset coefficient of the Y-axis.

[0176] Optionally, the master ring is worn on the thumb, with the default value of 'a' being 0.15 and the default value of 'b' being 0.16.

[0177] When the head control device and the main control ring are located on one side, the above two calculation methods ensure that the starting point of the aiming line is located in the middle of the hand and head, making it convenient for the user to operate.

[0178] S5022: Using the starting point as the origin, render a ray pointing towards the 3D interface according to the direction represented by the rotation data of the master control ring to obtain the aiming line.

[0179] When entering gesture interaction mode, the IMU sensor of the main control ring will collect its own rotation data in real time and store it in the storage chip. This rotation data can be sent to the head control device via Bluetooth connection.

[0180] When the head-mounted control device is integrated into the head-mounted display, it directly renders a ray pointing towards the 3D interface, using the starting point as the origin and following the direction represented by the rotation data of the main control ring, thus obtaining the aiming line. When the head-mounted control device is independent of the head-mounted display, the rotation data of the main control ring and the starting point are transmitted from the head-mounted control device to the head-mounted display via Bluetooth, so that the head-mounted display renders a ray pointing towards the 3D interface, using the starting point as the origin and following the direction represented by the rotation data, thus obtaining the aiming line.

[0181] Specifically, rotation data is represented in the form of quaternions. During the aiming line rendering process, the direction of the aiming line is obtained by multiplying the quaternion with the initialized rotation matrix. Then, with the starting point as the origin, a ray is rendered in that direction and directed toward the 3D interface to obtain the aiming line for interaction. The movement and rotation direction of the aiming line is consistent with the movement and rotation direction of the hand on which the ring is located, but the starting point position remains fixed.

[0182] like Figure 7 The diagram shows the movement of the aiming line. The aiming line moves with the starting point as the origin, and moves up, down, left, and right as the hand moves.

[0183] In gesture interaction mode, the head control device determines the starting point and direction of the aiming line based on its own position data, the position data of the main control ring, and rotation data. This enables the joint calibration of the aiming line in the three-dimensional interface using both the head and hand, improving the stability and accuracy of the aiming line. At the same time, it allows users to select targets by moving the aiming line in the three-dimensional interface through finger movements, making the operation simple and convenient.

[0184] S503: After aiming at the target on the three-dimensional interface with the aiming line, the distance between the two fingers is determined based on the position data of the other party collected by the main control ring and the auxiliary control ring respectively, so as to realize the confirmation operation triggered by the pinching of the fingers.

[0185] Users control the aiming line in the 3D interface by moving the finger on which the main control ring is located. Once the aiming line is aimed at the target in the 3D interface, the user confirms the action by pinching the two fingers wearing the main control ring and the auxiliary control ring.

[0186] In gesture interaction mode, the ToF sensor of the main control ring monitors the position data of the auxiliary control ring in real time and stores this position data in the storage chip. Simultaneously, the ToF sensor of the auxiliary control ring also monitors the position data of the main control ring in real time, stores this position data in the storage chip, and sends it to the main control ring via Bluetooth. The main control ring then sends the position data stored in the storage chip (including its own collected position data and the position data received from the auxiliary control ring) to the head-mounted device via Bluetooth. The head-mounted device's computing chip executes the confirmation logic after aiming at the target based on the position data collected by the main and auxiliary control rings, such as... Figure 8 As shown, it mainly includes the following steps:

[0187] S5031: Determine the first position coordinates of the main control ring based on the fingertip position coordinates and at least one finger joint position coordinates of the hand where the main control ring is located, and determine the second position coordinates of the auxiliary control ring based on the fingertip position coordinates and at least one finger joint position coordinates of the hand where the auxiliary control ring is located.

[0188] The Time-of-Flight (ToF) sensor in the master ring can detect the coordinates of various points on the finger where the auxiliary ring is located. Combined with the hand model, the length of the finger joints can be obtained, thereby determining the coordinates of the fingertip and various joint points on the finger where the auxiliary ring is located. Similarly, the ToF sensor in the auxiliary ring can also detect the coordinates of the fingertip and various joint points on the finger where the master ring is located.

[0189] Taking the example of the main control ring being worn on the thumb and the auxiliary control ring being worn at the base of the fingers other than the thumb, assuming the coordinates of the thumb's fingertip are (x... tt y tt , z tt The position coordinates of the intermediate joint are (x) tj y tj , z tj The coordinates of the fingertips of the other fingers are (x... ft y ft , z ft The coordinates of the first joint are (x) fj1 y fj1 , z fj1 ), second joint position coordinates (x) fj2 y fj2 , z fj2 At this point, the first position coordinates of the master control ring (i.e., the thumb) are calculated as follows:

[0190] x t =(x tt *c1+x tj *c2) / (c1+c2) Formula 7

[0191] y t =(y tt *c1+y tj *c2) / (c1+c2) Formula 8

[0192] z t =(z tt *c1+z tj *c2) / (c1+c2) Formula 9

[0193] The second position coordinates of the auxiliary control ring (i.e., other fingers) are calculated as follows:

[0194] x f =(x ft *d1+x fj1 *d2+x fj2 Formula 10: *d3) / (d1+d2+d3)

[0195] y f =(y ft *d1+y fj1 *d2+y fj2 Formula 11: *d3) / (d1+d2+d3)

[0196] z f =(z ft *d1+z fj1 *d2+z fj2 Formula 12: *d3) / (d1+d2+d3)

[0197] Where c1, c2, d1, d2, and d3 are weighted average parameters.

[0198] Optional, c1=2, c2=1, d1=6, d2=3, d3=1.

[0199] In the embodiments of this application, by wearing the main control ring on the thumb and the auxiliary control ring on other fingers, the distance between the two fingers can be increased, thereby avoiding misjudgment of gestures with small finger distances and improving interaction accuracy. On the other hand, by calculating the position of the ring using the fingertip position coordinates and at least one finger joint position coordinates, the accuracy of the two-finger distance calculation is improved to enhance the judgment of pinch operations, thereby improving interaction accuracy.

[0200] It should be noted that the embodiments of this application do not impose restrictive requirements on the weighted average parameters, which can be adjusted according to actual applications.

[0201] S5032: Determine the distance between the two fingers based on the first position coordinates and the second position coordinates, and determine whether a confirmation operation triggered by finger pinching has occurred based on the comparison result between the distance between the two fingers and the preset distance threshold.

[0202] Taking the example of the main control ring being worn on the thumb and the auxiliary control ring being worn at the base of other fingers besides the thumb, the distance between the two fingers is calculated as follows:

[0203] L = sqrt[(x t -x f ) 2 +(y t -y f ) 2 +(z t -z f ) 2 ] Formula 13

[0204] When the head-controlled device is integrated into a head-mounted display, it directly compares the distance between two fingers with a preset distance threshold and determines whether a pinch-to-close confirmation operation has occurred based on the comparison result. When the head-controlled device is independent of the head-mounted display, it sends the determined two-finger distance to the head-mounted display via Bluetooth, allowing the display to determine whether a pinch-to-close confirmation operation has occurred based on the comparison result.

[0205] Specifically, if the distance between the two fingers is less than a preset distance threshold, and the time that the distance is less than the preset distance threshold exceeds a preset time threshold, the pinch operation is considered successful, and the target is selected or opened. Otherwise, the pinch operation is considered to have failed, and the user has not issued a confirmation operation for the selected target.

[0206] like Figure 9 As shown, this is a schematic diagram of the gesture interaction operation provided in the embodiment of this application. During the gesture interaction, the movement of the hand wearing the main control ring and the auxiliary control ring will drive the movement of the aiming line in the display interface to aim at the target. When the distance between the two fingers of the finger wearing the main control ring and the finger wearing the auxiliary control ring is less than the preset distance threshold, it can be determined that the pinch operation is successful and the user has performed a confirmation operation of selecting or opening the target.

[0207] In gesture interaction mode, since the main control ring and the auxiliary control ring are worn on two fingers of one hand, the distance between the two fingers can be determined by the position coordinates of the two fingers. Based on the comparison between the distance between the two fingers and a preset distance threshold, it can be determined whether the user has performed a pinch operation, thus realizing gesture interaction of the head-mounted display device.

[0208] S504: Determine the aiming line in the three-dimensional interface based on the fingertip position data collected by any ring.

[0209] In desktop touch interaction mode, the ring's sensor has a certain detection range. When the main control ring and auxiliary control ring are placed on the table, this detection range forms a certain touch operation area on the table. The ToF sensor of the main control ring can detect the position data of the left or right fingertips within the touch operation area, and the ToF sensor of the auxiliary control ring can also detect the position data of the left or right fingertips within the touch operation area. Therefore, the head control device can determine the aiming line in the 3D interface based on the fingertip position data collected by either ring. See details for further information. Figure 10 It mainly includes the following steps:

[0210] S5041: Determine the touch calibration point on the Overlay calibration plane based on the fingertip position data collected by any ring.

[0211] The ToF sensor in the head-mounted control device monitors positional changes of the head-mounted display, while the IMU sensor monitors rotational changes. Upon entering desktop touch interaction mode, the head-mounted display creates an Overlay calibration plane based on the position and rotation data monitored by the head-mounted control device. The aspect ratio of this Overlay calibration plane can be set to the same as the aspect ratio of the touch operation area formed by the ring placed on the desktop. The position of the main control ring or auxiliary control ring is mapped to the origin of this Overlay calibration plane. The starting point of the aiming line is located on the Overlay calibration plane and is denoted as the touch calibration point. The direction of the aiming line is always perpendicular to the 3D interface. Figure 11 The figure shows a schematic diagram of the coordinate axes of the Overlay calibration plane.

[0212] When the user's head moves, the overlay calibration plane maintains consistent position and orientation with the 3D interface based on changes in position and rotation data collected by the head control device, and its area is consistent with the area of ​​the 3D interface. This overlay calibration plane is a transparent image, located above and parallel to the 3D interface.

[0213] After the main or auxiliary control ring is placed on the table, it can collect positional data such as the distance and direction of the user's left or right fingertips, and transmit this positional data to the head-control device in real time via Bluetooth. The head-control device's computing chip uses this positional data to calculate the mapping relationship between the finger's touch position and the position of the aiming line on the display screen, thereby determining the touch calibration point on the overlay calibration plane. The mapping process of the touch calibration point is as follows: Figure 12 As shown, it mainly includes the following steps:

[0214] S5041_1: Based on the distance from the fingertip to the corresponding ring and the aspect ratio within the touch operation area of ​​any ring, determine the distance between the mapping point on the Overlay calibration plane and the origin of the coordinate system.

[0215] In this system, the aspect ratio of the touch operation area of ​​the ring placed on the desktop is the same as that of the Overlay calibration plane. In desktop touch interaction mode, the starting point of the aiming line in the 3D interface switches to the Overlay calibration plane. That is, there is a mapping point of the starting point on the Overlay calibration plane. The head control device can determine the distance between this mapping point and the origin of the coordinate system based on the distance from the fingertip to the corresponding ring and the aspect ratio of the touch operation area of ​​any ring.

[0216] For example, assuming that the aspect ratio of the touch operation area and the aspect ratio of the Overlay calibration plane are both 1:5, when the ToF sensor measures the distance from the fingertip to the corresponding ring as d, the mapping point on the Overlay calibration plane is located at a distance of 5d from the origin.

[0217] S5041_2: Determine the direction of the mapping point on the Overlay calibration plane based on the direction of the fingertip relative to the finger ring within the touch operation area of ​​any finger ring.

[0218] Since the ring is located at the origin of the Overlay calibration plane, and the aspect ratio of the touch operation area is the same as that of the Overlay calibration plane, the direction of the fingertip relative to the ring within the touch operation area is the direction of the mapped point on the Overlay calibration plane.

[0219] S5041_3: Determine the touch calibration points on the Overlay calibration plane based on the distance and direction of the mapping points.

[0220] After determining the distance and direction of the mapping point on the Overlay calibration plane, the touch calibration point corresponding to the aiming line in the 3D interface on the Overlay calibration plane can be obtained.

[0221] S5042: Using the touch calibration point as the origin, render a ray perpendicular to the 3D interface to obtain the aiming line.

[0222] When the head-mounted control device is integrated into the head-mounted display, it directly renders a ray perpendicular to the 3D interface, using the touch calibration point as the origin, to obtain the aiming line. When the head-mounted control device is independent of the head-mounted display, it sends the position data of the touch calibration point to the head-mounted display. The head-mounted display then renders an aiming line perpendicular to the 3D interface, starting from the touch calibration point. The intersection of the aiming line and the 3D interface is the aiming point.

[0223] The embodiments of this application switch the starting position of the aiming line to the touch calibration point on the created Overlay calibration plane. Since the aspect ratio of the touch operation area of ​​the ring is the same as that of the Overlay calibration plane, the movement of the fingertip in the touch operation area can drive the movement of the aiming line in the three-dimensional interface, thereby realizing desktop touch operation.

[0224] S505: After aiming the line of sight at the target on the three-dimensional interface, determine the amount of rotation change based on the rotation data collected by the corresponding ring to achieve the confirmation operation triggered by tapping the table.

[0225] In desktop touch interaction mode, users can slide their fingers on the corresponding touch operation area of ​​the main control ring or auxiliary control ring on the desktop. When the user's finger position changes, the position of the touch calibration point will also change, thereby causing the position of the aiming ray in the three-dimensional interface to change in order to aim at the target.

[0226] The main control ring's IMU sensor detects finger rotation data, as do the auxiliary control ring's IMU sensor. Once the aiming reticle is aligned with the target, the user sends a confirmation command by tapping the table with their fingertip. This tap causes the table to vibrate, and the pressure from this vibration acts on the ring, changing the acceleration of the ring's IMU data, thus enabling the confirmation operation in the desktop touch interaction mode. The specific process is as follows... Figure 13 As shown, it mainly includes the following steps:

[0227] S5051: Determine the amount of rotational change based on the acceleration collected by the corresponding ring.

[0228] The IMU sensor of the main control ring or auxiliary control ring sends the acceleration characterizing the table vibration to the head control device, which then obtains the rotational change based on the acceleration change.

[0229] S5052: Based on the comparison between the rotation change and the preset rotation threshold, determine whether a confirmation operation triggered by tapping the desktop has occurred.

[0230] When the head-controlled device is integrated into the head-mounted display device, it directly compares the rotation change with a preset rotation threshold and determines whether a confirmation operation of tapping the table has occurred based on the comparison result. When the head-controlled device is independent of the head-mounted display device, it sends the rotation change to the head-mounted display device via Bluetooth. The head-mounted display device compares the rotation change with a preset rotation threshold and determines whether a confirmation operation of tapping the table has occurred based on the comparison result.

[0231] Specifically, when determining desktop tapping based on rotation change, if the rotation change is greater than a preset rotation threshold, it is determined that the user has performed a tapping confirmation operation on the desktop, thereby selecting or opening the target; otherwise, the desktop tapping is determined to have failed, and no confirmation response is given.

[0232] like Figure 14 The diagram shown is a schematic of desktop touch interaction operation provided in the embodiment of this application. During the desktop touch interaction, the user can use their right or left hand to slide in the touch operation area controlled by a single finger ring. The sliding of the finger will control the movement of the aiming line and aiming point in the three-dimensional interface. After aiming at the target, the user taps the desktop with their finger. In this way, after the main control finger ring or the auxiliary control finger ring detects the user's tapping action, the target aiming confirmation operation is completed.

[0233] In the embodiments of this application, in the desktop touch interaction mode where the main control ring or the auxiliary control ring is placed on the desktop, the head control device determines the aiming line using the Overlay calibration plane based on the position data of the fingertip corresponding to any ring, thereby completing the rendering of the aiming line in the three-dimensional interface with high aiming accuracy. On the other hand, the user changes the aiming line in the three-dimensional interface through the movement of the fingertip, which is convenient to operate and avoids arm fatigue caused by prolonged arm raising, effectively improving the user interaction experience.

[0234] In one example, when a user wants to end the interaction, they can shut down the operating system via the power button on the head-controlled device. For instance, after pressing the power button on the head-controlled device three times in quick succession, the head-controlled device shuts down. Consequently, the Bluetooth connection between the main control ring and the head-controlled device fails, and the main control ring automatically shuts down after the Bluetooth connection fails. Similarly, the Bluetooth connection between the auxiliary control ring and the main control ring also automatically shuts down after the Bluetooth connection fails.

[0235] In the embodiments of this application, two interaction modes are implemented based on the two usage methods of the main control ring and the auxiliary control ring. One is to wear it on two fingers of one hand to realize the gesture interaction mode, and the other is to place it on a table to realize the desktop touch interaction mode. The two interaction modes are easy to switch. In this way, during interaction, the user can trigger the indication information by operating the main control ring according to the actual interaction needs. The head control device determines the interaction mode according to the indication information, realizing flexible selection of the interaction mode in different scenarios, making the operation of the head-mounted display device more flexible and diverse, and the application scenarios more extensive.

[0236] For example, in a standing position, users can wear the main control ring and the auxiliary control ring on their fingers to enter the gesture interaction mode; in a sitting position, users can place the main control ring and the auxiliary control ring on the table to enter the desktop touch interaction mode.

[0237] See Figure 15The following is a timing diagram of the interaction between the operating system and the head-mounted display device provided in the embodiments of this application, which mainly includes the following steps:

[0238] S1: After the power switches of the head-mounted display device, head control device, main control ring, and auxiliary control ring are turned on, the head control device connects to the head-mounted display device via Bluetooth, the main control ring connects to the head control device via Bluetooth, and the auxiliary control ring connects to the main control ring via Bluetooth.

[0239] The Bluetooth connection is used to transmit data collected during the interaction process. The pairing process is described in the above embodiment and will not be repeated here.

[0240] S2: The operating system defaults to the gesture interaction mode. The head control device is fixed to one side of the head-mounted display device, and the main control ring and auxiliary control ring are worn on the thumb and other fingers of the corresponding hand on that side, respectively.

[0241] By wearing the master control ring and the auxiliary control ring on the thumb and other fingers of one hand, the gap between the master control ring and the auxiliary control ring can be increased, thereby improving the accuracy of gesture recognition.

[0242] S3: The ToF sensor of the head control device collects its own position data and the position data of the main control ring.

[0243] The position data collected by the head-controlled device can be stored on the head-controlled device's storage chip for subsequent calculations.

[0244] S4: The head-controlled device's computing chip determines the starting point of the aiming line based on its own position data and the position data of the main control ring, and sends it to the head-mounted display device via Bluetooth.

[0245] The head-controlled device's storage chip stores its own position data and the main control ring's position data, which are then transmitted to the computing chip to calculate the starting point of the aiming line. The specific calculation process is detailed in the aforementioned embodiment and will not be repeated here.

[0246] S5: The IMU sensor of the main control ring collects the rotation data of the thumb in real time and sends the rotation data to the head control device via Bluetooth. The head control device then sends the data to the head-mounted display device via Bluetooth.

[0247] The rotation data is represented in quaternion form and is used by head-mounted display devices to determine the direction of the aiming line.

[0248] S6: The head-mounted display device uses the starting point as the origin and renders a ray pointing towards the three-dimensional interface according to the direction represented by the rotation data to obtain the aiming line.

[0249] Since the aiming line is determined using the position and rotation data of the master control ring, the user can control the movement of the aiming line by hand movements to aim at the target in the three-dimensional interface.

[0250] S7: The ToF sensor of the main control ring collects the position data of the auxiliary control ring, and the ToF sensor of the auxiliary control ring collects the position data of the main control ring and sends it to the head control device.

[0251] In gesture mode interaction, the ToF sensor of the main control ring collects the location data of the auxiliary control ring in real time and stores it in the storage chip. At the same time, the ToF sensor of the auxiliary control ring collects the location data of the main control ring in real time and sends it to the main control ring via Bluetooth. The main control ring then sends the location data stored in the storage chip (including the location data it collects and the location data received from the auxiliary control ring) to the head-mounted device via Bluetooth.

[0252] S8: The head control device determines the distance between two fingers based on the position data of the auxiliary control ring and the main control ring, and sends the distance between the two fingers to the head-mounted display device via Bluetooth connection.

[0253] The calculation process for the distance between the two fingers is described in the aforementioned embodiment and will not be repeated here.

[0254] S9: The head-mounted display device compares the distance between two fingers with a preset distance threshold and determines whether a confirmation operation triggered by finger pinching has occurred based on the comparison result.

[0255] Specifically, when the distance between the two fingers is less than a preset distance threshold, and the time that the distance is less than the preset distance threshold exceeds a preset time threshold, the pinch operation is considered successful, and the target is selected or opened. Otherwise, the pinch operation is considered to have failed, and the user has not issued a confirmation operation for the selected target, thus realizing gesture interaction.

[0256] S10: A short press of the power button on the head control device switches the head control device, main control ring, and auxiliary control ring to desktop touch interaction mode.

[0257] When switching to desktop touch interaction mode, the head control device remains fixed, and the main control ring and auxiliary control ring are placed on a flat desktop.

[0258] S11: The head control device collects posture data and sends it to the head-mounted display device.

[0259] The pose data includes position data collected by the ToF sensor and rotation data collected by the IMU sensor, which are transmitted to the head-mounted display device via Bluetooth.

[0260] S12: The head-mounted display device creates an overlay calibration plane based on the received pose data.

[0261] The head-mounted display device creates an Overlay calibration plane based on the received pose data. The position of the main control ring or auxiliary control ring is mapped to the origin of this Overlay calibration plane. At this time, the starting point of the aiming line is located on the Overlay calibration plane and is recorded as the touch calibration point. The direction of the aiming line is always perpendicular to the three-dimensional interface.

[0262] S13: The main control ring or auxiliary control ring collects the position data of the fingertip and sends it to the head control device.

[0263] S14: The head control device determines the touch calibration point on the Overlay calibration plane based on the fingertip position data and sends it to the head-mounted display device.

[0264] The process of determining the touch calibration point is described in the aforementioned embodiment and will not be repeated here.

[0265] S15: The head-mounted display device uses the touch calibration point as the origin to render a ray perpendicular to the three-dimensional interface to obtain the aiming line.

[0266] Since the aiming line is determined using fingertip position data, users can slide their fingers on the corresponding touch operation area of ​​the main control ring or auxiliary control ring on the desktop to change the position of the touch calibration point, thereby causing the position of the aiming ray in the three-dimensional interface to change in order to aim at the target.

[0267] S16: The IMU sensor of the main control ring or auxiliary control ring collects the rotation data of the operator's hand and sends it to the head control device.

[0268] S17: The main control device determines the amount of rotation change based on the acceleration of the corresponding ring and sends it to the head-mounted display device.

[0269] Before and after a finger taps the table, the angular velocity of the ring on the finger changes, so the change in rotation can be used to determine whether an interaction occurs.

[0270] S18: The head-mounted display device compares the amount of rotation change with a preset rotation threshold and determines whether a confirmation operation triggered by tapping the desktop occurs based on the comparison result.

[0271] Specifically, if the rotation change is greater than the preset rotation threshold, it is determined that the user has performed a tap on the desktop to confirm the action, thereby selecting or opening the target. Otherwise, it is determined that the tap on the desktop failed and no confirmation response is given, thus realizing desktop touch interaction.

[0272] S19: Turn off the power button on the head control device, and the Bluetooth connection between the head control device and the main control ring will be disconnected.

[0273] When the user wants to end the interaction, they can press the power button on the head control device to turn it off. After the head control device is turned off, its Bluetooth connection with the main control ring is disconnected.

[0274] S20: The Bluetooth connection between the auxiliary control ring and the main control ring is disconnected, and both the main control ring and the auxiliary control ring are turned off.

[0275] In the embodiments of this application, the operating system comprised of the head-controlled device, the main control ring, and the auxiliary control ring is independent of the head-mounted display device. It is small in size, allowing for quick installation during interaction and rapid removal afterward, thereby reducing the interaction burden caused by the weight and size of the head-mounted display device. The main control ring and auxiliary control ring can be worn on a finger or placed on a table, providing both gesture interaction and desktop touch interaction modes for the head-mounted display device, making its operation more flexible and diverse. In gesture interaction mode, the head-controlled device determines the aiming line in the 3D interface based on its own position data and the pose data of the main control ring. Since the aiming line is determined using data from both the head and hand, the aiming process in the 3D interface is more stable and accurate, thus improving the interaction precision. Furthermore, after the aiming line is aimed at the target in the 3D interface by hand movement, the distance between the two fingers is determined based on the position data of the target collected by the main and auxiliary control rings to determine whether the user has confirmed the target, thus realizing gesture interaction. In desktop touch interaction mode, the head-controlled device determines the aiming line in the 3D interface based on the fingertip position data collected by the ring. After the aiming line is aimed at the target by sliding the finger on the desktop, the amount of rotation change is determined based on the rotation data collected by the corresponding ring to determine whether the user has confirmed the target, thus realizing desktop touch interaction.

[0276] Based on the same technical concept, this application provides a head control device that can implement the steps of the above-mentioned three-dimensional interface operation method and achieve the same technical effect.

[0277] The structure of the head control device is as follows: Figure 2 As shown, it includes a second pose sensor, a second Bluetooth module, a second storage chip, and a computing chip;

[0278] The second pose sensor is used to collect position and rotation data;

[0279] The second Bluetooth module is used to connect with the main control ring via Bluetooth, and the main control ring connects with the auxiliary control ring via Bluetooth.

[0280] The second memory chip includes a data storage area and a program storage area. The data storage area is used to store the data collected by itself and the data collected by the ring. The program storage area is used to store the computer program.

[0281] The computing chip is used to execute the steps of any three-dimensional interface operation method according to a computer program.

[0282] Based on the same technical concept, this application provides an operating system that can implement the steps of the above-described three-dimensional interface operation method and achieve the same technical effect.

[0283] The structure of the operating system is as follows Figure 2 As shown, it includes a head control device, a main control ring, and an auxiliary control ring, wherein:

[0284] The main control ring and the auxiliary control ring each contain a first posture sensor, a first Bluetooth module, a first storage chip, and a mode button, wherein:

[0285] The first attitude sensor is used to collect rotational and positional data, and the data is stored through the first storage chip.

[0286] The first Bluetooth module is used for Bluetooth connections between rings and Bluetooth connections with the head control device.

[0287] The mode button is used to send an instruction to the head control device to confirm the interaction mode in response to user operation;

[0288] The head control device includes a second pose sensor, a second Bluetooth module, a second storage chip, and a computing chip.

[0289] The second pose sensor is used to collect position and rotation data;

[0290] The second Bluetooth module is used for Bluetooth connection with the main control ring;

[0291] The second memory chip includes a program storage area and a data storage area. The data storage area is used to store the data collected by itself and the data collected by the ring.

[0292] The computing chip is used to execute the steps of any three-dimensional interface operation method according to the computer program stored in the program storage area.

[0293] In this embodiment, the memory chip can be volatile memory, such as random-access memory (RAM); it can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory chip can be a combination of the above-described memories.

[0294] A computing chip may include one or more central processing units (CPUs), GPUs, or digital processing units, etc. The computing chip is used to implement the operation methods of any of the aforementioned three-dimensional interfaces when calling computer programs stored in the memory chip.

[0295] In the embodiments of this application, the internal components of the head control device, the main control ring, and the auxiliary control ring can be connected via a bus, which can be divided into an address bus, a data bus, a control bus, etc.

[0296] For ease of description, the operating system can be divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0297] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0298] Based on the same technical concept, this application provides a head-mounted display device, which includes a head control device capable of implementing the above-mentioned three-dimensional interface operation method, and can achieve the same technical effect.

[0299] This application also provides a computer-readable storage medium for storing instructions that, when executed, can complete the steps of any of the three-dimensional interface operation methods described in the foregoing embodiments.

[0300] This application also provides a computer program product for storing a computer program that performs the steps of any of the three-dimensional interface operation methods described in the foregoing embodiments.

[0301] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0302] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0303] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0304] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0305] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for operating a three-dimensional interface, characterized in that, The method, applied to a head-controlled device, wherein the head-controlled device is connected to a main control ring via Bluetooth, and the main control ring is connected to an auxiliary control ring via Bluetooth, includes: The interaction mode is determined based on the indication information sent by the main control ring; wherein, the indication information is triggered based on user operation. When the interaction mode is gesture interaction mode, the starting point of the aiming line is determined based on its own position data and the position data of the main control ring. The starting point is located between the head and the hand. Taking the starting point as the origin, a ray is rendered towards the three-dimensional interface according to the direction represented by the rotation data of the main control ring to obtain the aiming line. After the aiming line is aimed at the target on the three-dimensional interface, the distance between the two fingers is determined based on the position data of the other party collected by the main control ring and the auxiliary control ring to realize the confirmation operation triggered by finger pinching. In the gesture interaction mode, the main control ring and the auxiliary control ring are worn on two fingers of one hand. When the interaction mode is desktop touch interaction mode, a touch calibration point on the Overlay calibration plane is determined based on the fingertip position data collected by any ring. Using the touch calibration point as the origin, a ray perpendicular to the 3D interface is rendered to obtain the aiming line. After the aiming line is aimed at the target on the 3D interface, the rotation change is determined based on the rotation data collected by the corresponding ring to achieve a confirmation operation triggered by tapping the desktop. In the desktop touch interaction mode, the main control ring and the auxiliary control ring are placed on the desktop. The Overlay calibration plane is a transparent image, located on top of the 3D interface and parallel to it. The origin of the Overlay calibration plane is the position of the ring, and the aspect ratio of the Overlay calibration plane is the same as the aspect ratio of the touch operation area of ​​the ring on the desktop.

2. The method as described in claim 1, characterized in that, When the main control ring is worn on the left hand, the starting point position is calculated as follows: When the main control ring is worn on the right hand, the starting point position is calculated as follows: in,( , , ) represents the position coordinates of the head control device. , , () represents the position coordinates of the main control ring. This is the offset coefficient along the X-axis. This is the offset coefficient for the Y-axis.

3. The method as described in claim 1, characterized in that, The step of determining the distance between two fingers based on the position data of the other party collected by the main control ring and the auxiliary control ring respectively to realize the confirmation operation triggered by finger pinching includes: The first position coordinates of the main control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the main control ring is located; and the second position coordinates of the auxiliary control ring are determined based on the fingertip position coordinates and at least one finger joint position coordinates of the hand in which the auxiliary control ring is located. Based on the first position coordinates and the second position coordinates, the distance between the two fingers is determined, and based on the comparison result of the distance between the two fingers and a preset distance threshold, it is determined whether a confirmation operation triggered by finger pinching has occurred.

4. The method as described in claim 3, characterized in that, When the main control ring is worn on the thumb and the auxiliary control ring is worn on other fingers besides the thumb, the first position coordinates are calculated as follows: The second position coordinate is calculated as follows: in,( , , ) represents the coordinates of the fingertip position of the thumb. , , ( ) represents the coordinates of the middle joint of the thumb; , , ) represents the coordinates of the fingertip positions of the other fingers, ( , , ) represents the coordinates of the first joint position of the other fingers, ( , , () represents the coordinates of the second joint position of the other fingers. , , , and The parameter is the weighted average parameter.

5. The method as described in claim 1, characterized in that, The aspect ratio of the touch operation area of ​​the ring placed on the desktop is the same as the aspect ratio of the Overlay calibration plane. The step of determining the touch calibration point on the Overlay calibration plane based on the fingertip position data collected by any ring includes: Based on the distance from the fingertip to the corresponding ring within the touch operation area of ​​any ring and the aspect ratio, determine the distance between the mapping point on the Overlay calibration plane and the origin of the coordinate system. The orientation of the mapping point on the Overlay calibration plane is determined based on the orientation of the fingertip relative to the ring within the touch operation area of ​​any ring; Based on the distance and direction corresponding to the mapping point, the touch calibration point on the Overlay calibration plane is determined.

6. The method as described in claim 1, characterized in that, The step of determining the amount of rotation change based on the rotation data collected by the corresponding ring to achieve the confirmation operation triggered by tapping the desktop includes: The amount of rotational change is determined based on the acceleration collected by the corresponding ring; wherein, the change in acceleration is generated when the corresponding ring is subjected to pressure, and the pressure is generated by the vibration of the table caused by the fingertip tapping the table. Based on the comparison between the rotation change and the preset rotation threshold, it is determined whether a confirmation operation triggered by tapping the desktop has occurred.

7. A head control device, characterized in that, It includes a pose sensor, a Bluetooth module, a storage chip, and a computing chip; The Bluetooth module is used to connect with the main control ring via Bluetooth, and the main control ring is connected with the auxiliary control ring via Bluetooth. The storage chip includes a data storage area and a program storage area. The data storage area is used to store data collected by itself and data collected by the ring. The program storage area is used to store computer programs. The computing chip performs the following operations according to the computer program: The Bluetooth module receives instruction information sent by the master control ring and determines the interaction mode based on the instruction information; wherein the instruction information is triggered based on user operation. When the interaction mode is gesture interaction mode, the starting point of the aiming line is determined based on its own position data and the position data of the main control ring. The starting point is located between the head and the hand. Taking the starting point as the origin, a ray is rendered towards the three-dimensional interface according to the direction represented by the rotation data of the main control ring to obtain the aiming line. After the aiming line is aimed at the target on the three-dimensional interface, the Bluetooth module receives the position data of the other party sent by the main control ring and the auxiliary control ring, and determines the distance between the two fingers based on the other party's position data to realize the confirmation operation triggered by finger pinching. In the gesture interaction mode, the main control ring and the auxiliary control ring are worn on two fingers of one hand. When the interaction mode is desktop touch interaction mode, the Bluetooth module receives the fingertip position data collected by any ring to determine the touch calibration point on the Overlay calibration plane. With the touch calibration point as the origin, a ray perpendicular to the three-dimensional interface is rendered to obtain the aiming line. After the aiming line is aimed at the target on the three-dimensional interface, the Bluetooth module receives the rotation data collected by the corresponding ring and determines the rotation change based on the rotation data to realize the confirmation operation triggered by tapping the desktop. In the desktop touch interaction mode, the main control ring and the auxiliary control ring are placed on the desktop. The Overlay calibration plane is a transparent screen, located on the upper layer of the three-dimensional interface and parallel to the three-dimensional interface. The coordinate origin of the Overlay calibration plane is the position of the ring. The aspect ratio of the Overlay calibration plane is the same as the aspect ratio of the touch operation area of ​​the ring on the desktop.

8. A three-dimensional interface operating system, characterized in that, Includes a head control device, a main control ring, and an auxiliary control ring; The main control ring and the auxiliary control ring each include a first posture sensor, a first Bluetooth module, a first storage chip, and a mode button, wherein: The first pose sensor is used to collect rotation data and position data, and stores the data through the first storage chip; The first Bluetooth module is used for Bluetooth connection between the rings and Bluetooth connection with the head control device; The mode button is used to send an instruction message to the head control device to determine the interaction mode in response to user operation; The head control device includes a second pose sensor, a second Bluetooth module, a second storage chip, and a computing chip. The second pose sensor is used to collect position and rotation data; The second Bluetooth module is used to establish a Bluetooth connection with the main control ring; The second storage chip includes a program storage area and a data storage area, wherein the data storage area is used to store data collected by itself and data collected by the ring; The computing chip is used to execute the method of any one of claims 1 to 6 according to the computer program stored in the program storage area.

Images