Head-mounted display device, interaction system and interaction method

Abstract

The invention relates to the technical field of terminals, in particular to head-mounted display equipment, an interaction system and an interaction method. The interactive system comprises a head display device and a handle which are in communication connection. The head display device is used for displaying a display interface. The handle comprises an inertia measurement module and a calibration module, the inertia measurement module is used for measuring inertia data when the handle moves, the calibration module is used for measuring calibration data when the handle moves, and the calibration data is position or angle data with the earth as a reference system when the handle moves. The head-mounted display device can obtain the displacement data when the handle moves, and determine the position of a cursor in the display interface according to the displacement data of the handle. Wherein the displacement data of the handle is obtained by calibrating the inertial data according to the calibration data. The cost of the interaction system can be effectively reduced, and the miniaturization design of the handle can be realized.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a head-mounted display device, an interaction system, and an interaction method. Background Technology

[0002] With the development of science and technology, users can interact with various head-mounted display devices to obtain a fully immersive experience. For example, in extended reality (XR) scenarios such as virtual reality (VR), augmented reality (AR), and augmented virtuality (AV), or mixed reality (MR) scenarios, users can interact with virtual game scenes constructed by head-mounted display devices to obtain an immersive gaming experience.

[0003] In some solutions, users can interact with the virtual scene constructed by the head-mounted display device through eye movements and hand pinching. However, this interaction method requires slow, step-by-step execution; for example, the eyes must first gaze before the hand pinches to ensure that the head-mounted display device can accurately recognize the user's movements. Therefore, it can only meet the needs of slow-paced and simple operations, and its applicability is limited.

[0004] Therefore, in some alternative solutions, users can interact with the head-up display (HUD) using controllers to achieve a better interactive experience. However, this interaction requires determining the controller's position relative to the HUD through optical positioning, which is costly and makes it difficult to reduce the size of the controller. Summary of the Invention

[0005] This application provides a head-mounted display device, an interaction system, and an interaction method. The following describes this application from multiple aspects, and the implementation methods and beneficial effects of these aspects can be referenced each other.

[0006] The first aspect of this application provides an interactive system. Specifically, the interactive system includes a head-mounted display (HUD) and a controller connected in communication. The HUD is used to display a user interface. The controller includes an inertial measurement module and a calibration module. The inertial measurement module measures the inertial data when the controller moves, and the calibration module measures the calibration data when the controller moves. The calibration data is the position or angle data of the controller relative to the Earth as a reference frame. The HUD can acquire the displacement data when the controller moves and determine the position of the cursor on the display interface based on the displacement data. The displacement data of the controller is obtained by calibrating the inertial data based on the calibration data.

[0007] According to the embodiments of this application, the displacement data of the handle itself during movement is determined by setting an inertial measurement module and a calibration module on the handle, rather than by setting a dedicated camera on the head-mounted display device to track the optical positioning of the optical devices on the handle to determine the position of the handle relative to the head-mounted display device. In this way, there is no need to set a camera module on the head-mounted display device for optical positioning, thereby effectively reducing costs. Furthermore, there is no need to set optical devices (e.g., infrared light sources, visible light sources, etc.) on the handle for optical positioning, which is beneficial for miniaturizing the handle design.

[0008] In one possible implementation of the first aspect described above, the calibration module includes an electronic compass, and the calibration data includes geomagnetic declination data.

[0009] According to an embodiment of this application, an electronic compass is used to measure the geomagnetic declination data when the handle is moved. The geomagnetic declination data can provide azimuth information, effectively correct the errors accumulated during the integration process of the inertial measurement unit, and is low in cost.

[0010] In one possible implementation of the first aspect described above, the inertial measurement module includes multiple inertial measurement units, and the inertial data includes inertial data measured by the multiple inertial measurement units. By combining the inertial data measured by the multiple inertial measurement units, the measurement error can be further reduced, thereby improving the overall measurement accuracy.

[0011] In one possible implementation of the first aspect described above, the head-mounted display device is able to acquire inertial data and calibration data, and calibrate the inertial data based on the calibration data to obtain displacement data when the handle moves.

[0012] Since inertial and calibration data are processed through the head-mounted display, the controllers do not need to have high processing power, which can effectively reduce the cost of the controllers.

[0013] In one possible implementation of the first aspect described above, the head-mounted display device determining the position of the cursor on the display interface based on the displacement data of the handle includes: the head-mounted display device converting the displacement data of the handle into cursor displacement data; and the head-mounted display device determining the position of the cursor on the display interface based on the cursor displacement data. In this way, the cursor on the display interface can move accordingly with the movement of the handle, thereby achieving human-computer interaction.

[0014] In one possible implementation of the first aspect described above, the displacement data of the handle is obtained by converting the Cartesian coordinate system to the first spherical coordinate system. The head-mounted display device converts the displacement data of the handle into the displacement data of the cursor, including: the head-mounted display device amplifies the radial data of the handle's displacement data according to a preset exponential mapping relationship to obtain the radial data of the cursor; amplifies the azimuth data of the handle's displacement data according to a first preset proportional mapping relationship to obtain the azimuth data of the cursor; amplifies the polar angle data of the handle's displacement data according to a second preset proportional mapping relationship to obtain the polar angle data of the cursor; and uses the radial data, polar angle data, and radial data of the cursor as the displacement data of the cursor, wherein the displacement data of the cursor is data in the second spherical coordinate system.

[0015] According to the embodiments of this application, the radial data in the displacement data of the handle is amplified according to a preset exponential mapping relationship to obtain the radial data of the cursor. Therefore, in the radial direction, the greater the movement distance of the handle, the greater the corresponding movement distance of the cursor, which is exponentially increased. This better adapts to application scenarios involving interaction from near to far distances. In other words, when controlling the cursor to move radially to a greater distance, the handle does not need to move a proportionally larger distance radially, thus improving the user's control experience.

[0016] Based on the first preset proportional mapping relationship, the azimuth data in the handle's displacement data is amplified to obtain the cursor's azimuth data. Similarly, based on the second preset proportional mapping relationship, the polar angle data in the handle's displacement data is amplified to obtain the cursor's polar angle data. Thus, only a small movement of the handle is needed to achieve a large movement of the cursor, making operation simpler and more convenient.

[0017] In one possible implementation of the first aspect described above, the head-mounted display device includes a left lens and a right lens, the line connecting the center of the left lens and the center of the right lens is a first line, and the midpoint of the first line is the origin of a first spherical coordinate system.

[0018] In one possible implementation of the first aspect described above, the first spherical coordinate system is the same as the second spherical coordinate system.

[0019] In one possible implementation of the first aspect described above, the handle includes an origin return button, which is used to control the cursor in the display interface to return to the cursor origin in the display interface.

[0020] In this way, regardless of the cursor's position, clicking the "Return to Origin" button will return the cursor to its original position, enabling quick positioning, preventing cursor loss, and improving operational convenience. Furthermore, when the cursor moves abnormally, clicking the "Return to Origin" button will also return it to its original position for recalibration, restoring normal cursor movement.

[0021] In one possible implementation of the first aspect described above, the head-mounted display device includes a left lens and a right lens, with a first connecting line between the centers of the left and right lenses. A second connecting line is formed between the origin of the cursor and the midpoint of the first connecting line, and this second connecting line is perpendicular to the first connecting line and parallel to the optical axis of the left lens.

[0022] This allows the cursor origin to be roughly located at the visual center, thus improving the visual experience. For example, when the head-mounted display is in use, the user can see the cursor origin simply by looking straight ahead, without needing to look up, down, or to the left or right.

[0023] In one possible implementation of the first aspect mentioned above, the length of the second connection is 40cm to 120cm, for example, 40cm, 41cm, 42cm or 43cm, so that the cursor origin can be more easily seen when the head-mounted display is in the wearing state.

[0024] In one possible implementation of the first aspect mentioned above, the handle includes a 3D / 2D switching button, which is used to control the cursor's movement mode to switch between spatial movement mode and planar movement mode to adapt to different application scenarios and meet customized usage habits.

[0025] For example, in a game, the 3D / 2D switch button can be used to switch the cursor movement mode to spatial movement mode. In this mode, the cursor has three degrees of freedom and can move in three directions, allowing users to interact with any object in the game scene and achieve an immersive gaming experience. Similarly, in a web browsing scenario, the 3D / 2D switch button can be used to switch the cursor movement mode to planar movement mode. In this mode, the cursor has two degrees of freedom and can move in two directions, allowing users to browse web pages.

[0026] In one possible implementation of the first aspect mentioned above, the handle includes multiple buttons arranged in the shape of an outstretched five-finger grip to better match the user's finger position, resulting in better grip comfort and easier operation.

[0027] A second aspect of this application provides an interaction method. This interaction method is used for a head-mounted display device that is communicatively connected to a controller, and the head-mounted display device is used to display a display interface.

[0028] Specifically, the method includes: acquiring inertial data and calibration data when the handle moves, calibrating the inertial data according to the calibration data to obtain displacement data when the handle moves, wherein the calibration data is the position or angle data of the handle with the Earth as the reference frame when it moves; converting the displacement data when the handle moves into the displacement data of the cursor in the display interface; and determining the position of the cursor in the display interface according to the displacement data of the cursor.

[0029] In the above interaction method, the head-mounted display (HMD) determines the cursor position by acquiring the displacement data of the handle during movement, rather than acquiring the position of the handle relative to the HMD. This eliminates the need for a dedicated camera module for optical positioning of the handle, effectively reducing costs. Furthermore, the handle does not require additional optical components such as infrared or visible light sources for optical positioning, facilitating a miniaturized design.

[0030] In one possible implementation of the second aspect described above, the calibration data includes geomagnetic declination data.

[0031] According to an embodiment of this application, an electronic compass is used to measure the geomagnetic declination data when the handle is moved. The geomagnetic declination data can provide azimuth information, effectively correct the errors accumulated during the integration process of the inertial measurement unit, and is low in cost.

[0032] In one possible implementation of the second aspect described above, the displacement data of the handle is obtained by converting the Cartesian coordinate system to the first spherical coordinate system. Converting the handle's displacement data into cursor displacement data includes: amplifying the radial data in the handle's displacement data according to a preset exponential mapping relationship to obtain the cursor's radial data; amplifying the azimuth data in the handle's displacement data according to a first preset proportional mapping relationship to obtain the cursor's azimuth data; amplifying the polar angle data in the handle's displacement data according to a second preset proportional mapping relationship to obtain the cursor's polar angle data; and using the cursor's radial data, polar angle data, and radial data as the cursor's displacement data, which is data in the second spherical coordinate system.

[0033] According to the embodiments of this application, the radial data in the displacement data of the handle is amplified according to a preset exponential mapping relationship to obtain the radial data of the cursor. Therefore, in the radial direction, the greater the movement distance of the handle, the greater the corresponding movement distance of the cursor, which is exponentially increased. This better adapts to application scenarios involving interaction from near to far distances. In other words, when controlling the cursor to move radially to a greater distance, the handle does not need to move a proportionally larger distance radially, thus improving the user's control experience.

[0034] Based on the first preset proportional mapping relationship, the azimuth data in the handle's displacement data is amplified to obtain the cursor's azimuth data. Similarly, based on the second preset proportional mapping relationship, the polar angle data in the handle's displacement data is amplified to obtain the cursor's polar angle data. Thus, only a small movement of the handle is needed to achieve a large movement of the cursor, making operation simpler and more convenient.

[0035] In one possible implementation of the second aspect described above, the head-mounted display device includes a left lens and a right lens, the line connecting the center of the left lens and the center of the right lens is a first line, and the midpoint of the first line is the origin of a first spherical coordinate system.

[0036] In one possible implementation of the second aspect described above, the first spherical coordinate system is the same as the second spherical coordinate system.

[0037] A third aspect of this application provides a computer-readable storage medium. This computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the interactive methods described in the second aspect and any possible implementation thereof.

[0038] A fourth aspect of this application provides a head-mounted display device. The head-mounted display device includes a memory and one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, execute the interaction methods described in the second aspect above and any possible implementation thereof.

[0039] It should be understood that the beneficial effects that can be achieved by the third and fourth aspects mentioned above can be referred to the beneficial effects of the first or second aspects of this application, and will not be repeated here. Attached Figure Description

[0040] Figure 1 This application illustrates an exemplary application scenario of an interactive system according to an embodiment of the present application;

[0041] Figure 2 Exemplary block diagrams of interactive systems employing optical positioning in some technical solutions are shown;

[0042] Figure 3 An exemplary block diagram of the interactive system in an embodiment of this application is shown;

[0043] Figure 4 This illustration shows a logical diagram of how handle movement is converted into cursor movement in an interactive system according to an embodiment of this application.

[0044] Figure 5A This illustrates one exemplary configuration of the spherical coordinate system in an embodiment of this application;

[0045] Figure 5B This illustrates a second exemplary configuration of the spherical coordinate system in an embodiment of this application.

[0046] Figure 6 An exemplary distribution of multiple buttons in the handle in an embodiment of this application is shown;

[0047] Figure 7 A flowchart of an interaction method according to an embodiment of this application is shown;

[0048] Figure 8 An exemplary structural block diagram of an XR head-mounted display device according to an embodiment of this application is shown. Detailed Implementation

[0049] To facilitate understanding of the technical solution of this application, the concepts involved in this application will be explained first.

[0050] Inertial Measurement Unit (IMU): An inertial measurement unit is used to measure the inertial data of an object during its movement, thereby determining the object's pose.

[0051] For example, an inertial measurement unit (IMU) may include a three-axis accelerometer and a three-axis gyroscope. The three axes of the three-axis accelerometer and gyroscope are aligned to form the O-XYZ coordinate system of the IMU. The origin of this O-XYZ coordinate system can be, for example, the measurement center of the IMU. The sensors in the IMU, such as the three-axis accelerometer and the three-axis gyroscope, are symmetrically arranged about the measurement center to ensure data consistency and accuracy. The three-axis accelerometer is used to detect the acceleration of an object in the X, Y, and Z axes during movement, and the three-axis gyroscope is used to detect the angular velocity of the object in the X, Y, and Z axes during movement. By integrating the acceleration and angular velocity in the X, Y, and Z axes, the pose of the object relative to itself in three-dimensional space can be determined, for example, the displacement data of the object during movement. This displacement data is data in a Cartesian coordinate system O-XYZ, such as (ΔX, ΔY, ΔZ).

[0052] Inertial measurement units (IMUs) accumulate errors during integration, leading to a significant discrepancy between the final detected pose and the actual pose of the object. Therefore, individual IMU measurements are generally used as auxiliary calibration devices for other positioning schemes. For example, in optical positioning schemes, an IMU can be placed on the object for auxiliary calibration.

[0053] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0054] This application provides an interactive system. The interactive system includes a head-mounted display (HUD) and a controller connected in communication. The HUD displays an interface to present an interactive virtual scene to the user. The user can control a cursor on the display interface using the controller, thereby interacting with the virtual scene. An exemplary description follows.

[0055] Figure 1 An exemplary application scenario of an interactive system 1 according to an embodiment of this application is shown. (Reference) Figure 1The interactive system 1 includes an XR head-mounted display device 10 and a controller 20 connected for communication. It should be noted that AR head-mounted displays, AV head-mounted displays, and VR head-mounted displays can all be collectively referred to as XR head-mounted displays. In some embodiments, the controller 20 can communicate with the XR head-mounted display device 10 via wireless communication methods such as Bluetooth (BT). Alternatively, in other alternative embodiments, the controller 20 can also communicate with the XR head-mounted display device 10 via wired communication methods such as cables; this application does not impose any limitations on this.

[0056] The XR headset 10 can display a display interface 11. The display interface 11 can be, for example, a... Figure 1 The area enclosed by the two dotted lines. The display interface 11 is used to show virtual objects to the user, such as a cursor 12 and a rectangular object 13.

[0057] It is understood that this application does not specifically limit the shape of cursor 12; cursor 12 can be an icon or graphic of any shape, size, and transparency. For example, in some embodiments, cursor 12 can be... Figure 1 The cursor 12 is shown as a finger icon. In other alternative implementations, the cursor 12 may also be an arrow, triangle, or dot.

[0058] Furthermore, it is understood that this application does not specifically limit the shape of the handle 20; the handle 20 can be a structure of any shape and size. For example, in some embodiments, the handle 20 can be... Figure 1 The sphere shown. In other alternative embodiments, the handle 20 may also be a cuboid, cylinder, or other irregular structure.

[0059] As the user moves the controller 20, the XR headset 10 can locate the controller 20 and then determine the position of the cursor 12 within the display interface 11 based on the position of the controller 20, ultimately enabling the user to interact with the virtual scene constructed by the XR headset 10. For example, Figure 1 In the illustrated embodiment, when the handle 20 is in the first position P1, the cursor 12 is in the second position P2. The user can move the handle 20 from the first position P1 to the third position P3, thereby moving the cursor 12 from the second position P2 to the fourth position P4 to select the rectangular object 13 and interact with it, for example, by zooming in, zooming out, or dragging the rectangular object 13.

[0060] Understandable. Figure 1In the illustrated embodiment, the XR head-mounted display device 10 is used as an example of a head-mounted display device, and the handheld spherical handle 20 is used as an example of a handle. However, this application is not limited to this; in other embodiments, the head-mounted display device and the handle can also be other interactive electronic devices. For example, the head-mounted display device can also be an MR head-mounted display device. The handle can also be a wristband handle, a pen handle, or other types of handles. For ease of description, the following will continue to use... Figure 1 The interactive scenario shown is presented as an example.

[0061] In some technical solutions, the position of the controller relative to the XR headset can be determined by optical positioning, and then the position of the cursor on the display interface can be determined based on the position of the controller relative to the XR headset, thus realizing human-computer interaction.

[0062] Specifically, Figure 2 An exemplary block diagram of an interactive system 1a employing optical positioning in some technical solutions is shown. (Reference) Figure 2 The XR head-mounted display device 10a includes a camera module 100a. The controller 20a includes an infrared light source 200a (light emitting diode, LED), a visible light source 210a, an inertial measurement unit 220a (inertial measurement unit, IMU), an input module 230, a communication module 240, and a processor 250.

[0063] The infrared light source 200a emits infrared light, and the visible light source 210a emits visible light. For example, the infrared light source 200a and the visible light source 210a can be light-emitting diode (LED) strips. During the movement of the handle 20a, the camera module 100 of the XR headset 10a can capture the infrared light emitted by the infrared light source 200a and the visible light emitted by the visible light source 210a, thereby continuously observing the movement of the infrared light source 200a and the visible light source 210a, performing optical image recognition, and ultimately determining the position of the handle 20a relative to the XR headset 10a.

[0064] The inertial measurement unit 220a is used to measure the inertial data, such as angular velocity and acceleration, when the handle 20a moves, so as to assist in the calibration of the above-mentioned optical positioning scheme.

[0065] The input module 230 may include, for example, multiple buttons, allowing users to input different commands by clicking different buttons, thereby interacting with the virtual scene constructed by the XR head-mounted display device 10a.

[0066] The communication module 240 is used to enable communication between the handle 20a and the XR head-mounted display device 10a.

[0067] The processor 250 is used to control the overall operation of the handle 20a. For example, it controls the infrared light source 200a and the visible light source 210a to turn on or off, receives signals from the inertial measurement unit 220a and the buttons 230, and sends the signals to the XR head-mounted display device 10a via the communication module 260. In some embodiments, the processor 250 may be, for example, a microcontroller unit (MCU).

[0068] It is worth noting that in order to determine the position of the handle 20a relative to the XR headset 10a, a dedicated camera module 100a needs to be installed on the XR headset 10a to capture images of the handle 20a, thus increasing the equipment cost. Furthermore, the handle 20a needs to be equipped with an infrared light source 200a and a visible light source 210a, which occupy a significant amount of space, resulting in an excessively large size for the handle 20a.

[0069] To address the aforementioned problems, this application provides an interactive system. For ease of understanding, it will still be referred to as... Figure 1 The scenario shown in this application will be used as an example for explanation. In the interactive system provided in this application, the position of the handle relative to the XR headset is not determined by optical positioning. Instead, an inertial measurement module and a calibration module are installed on the handle to determine its displacement data during movement. This displacement data is then used to determine the cursor's position within the display interface, thus achieving human-computer interaction. Therefore, the XR headset does not need a dedicated camera module to capture images from the handle, effectively reducing costs. Furthermore, the handle does not need to have infrared or visible light sources or other optical components, allowing for a smaller size.

[0070] The following describes the interactive system 1 provided in this application. As described above, in this embodiment, the interactive system 1 includes an XR head-mounted display device 10 and a controller 20. The XR head-mounted display device 10 is used to display an interface 11. The XR head-mounted display device 10 and the controller 20 are communicatively connected. The user can control the position of the cursor 12 in the display interface 11 by moving the controller 20, thereby achieving human-computer interaction. The specific form and function of the display interface 11 and the cursor 12 can be referred to above. Figure 1 The relevant descriptions in the illustrated embodiments will not be repeated here. The following will focus on the handle 20 provided in the embodiments of this application.

[0071] Figure 3 An exemplary block diagram of the interactive system 1 in an embodiment of this application is shown. Continue to refer to... Figure 3 and combined Figure 1 The handle 20 includes an inertial measurement module 200 and a calibration module 210.

[0072] The inertial measurement module 200 is used to measure the inertial data of the handle 20 during its movement, such as acceleration and angular velocity. The inertial measurement module 200 may include one or more inertial measurement units (e.g., a first inertial measurement unit 201 and a second inertial measurement unit 202), which is not limited in this application.

[0073] The calibration module 210 is used to measure calibration data of the handle 20 during its movement. The calibration data is the position or angle data of the handle 20 relative to the Earth (e.g., magnetic declination data, latitude and longitude data, etc.) as it moves. In other words, the calibration data is measured relative to a fixed reference point on the Earth's surface or at the Earth's center, rather than relative to itself or other objects (e.g., the XR headset 10). The calibration data effectively helps to calibrate the accumulated errors during the integration process of the inertial measurement module 200, thereby improving the detection accuracy of the inertial measurement module 200.

[0074] By calibrating the inertial data measured by the inertial measurement module 200 based on the calibration data measured by the calibration module 210, the displacement data of the handle 20 during movement can be obtained. For example, in some embodiments, the calibration data and inertial data can be fused and calculated using fusion algorithms such as Kalman filters, complementary filters, and particle filters to obtain the displacement data of the handle 20 during movement. The XR head-mounted display device 10 can acquire the displacement data of the handle 20 during movement and determine the position of the cursor 12 within the display interface 11 based on this displacement data, ultimately realizing human-computer interaction.

[0075] Compared to the above Figure 2 The interactive system 1a shown in this application determines the displacement data of the handle 20 itself during movement by setting an inertial measurement module 200 and a calibration module 210 on the handle 20, rather than determining the position of the handle 20 relative to the XR head-mounted display device 10 through optical positioning. This eliminates the need for a dedicated camera module on the XR head-mounted display device 10 to capture images from the handle 20, effectively reducing costs. Furthermore, the handle 20 does not require additional optical components such as infrared or visible light sources, facilitating miniaturization. Simultaneously, the integrated positioning method of the inertial measurement module 200 and the calibration module 210 helps to accurately identify the displacement data of the handle 20 during movement, resulting in high detection accuracy.

[0076] The following section will continue to describe in detail the exemplary scheme by which the XR head-mounted display device 10 acquires the displacement data of the handle 20 during movement and determines the position of the cursor 12 based on the displacement data.

[0077] In some feasible solutions, the XR head-mounted display device 10 can acquire inertial data measured by the inertial measurement module 200 and calibration data measured by the calibration module 210, and calibrate the inertial data according to the calibration data to obtain the displacement data of the handle 20. Then, the displacement data of the handle 20 is amplified to obtain the displacement data of the cursor 12. Finally, the position of the cursor 12 is determined according to the displacement data of the cursor 12.

[0078] Thus, only a small movement of the controller 20 is needed to satisfy the large-scale movement requirements of the cursor 12, without having to raise the controller 20 to aim at objects in the virtual scene constructed by the XR headset 10, conforming to the usage habits of a traditional mouse, making operation simple and convenient. For example Figure 1 In the example shown, the user can move the handle 20 slightly, so that the handle 20 moves from the first position P1 to the third position P3. During this process, the cursor 12 can move significantly in response to the movement of the handle 20, so that the cursor 12 moves from the second position P2 to the fourth position P4, so as to select the rectangular object 13 and interact with the rectangular object 13, such as zooming in, zooming out or dragging the rectangular object 13.

[0079] Specifically, Figure 4 This illustration shows a logical diagram of how movement of the handle 20 is converted into movement of the cursor 12 in the interactive system 1 of this application embodiment. For example, refer to... Figure 4 and combined Figure 1 When the handle 20 is in the first position P1, the cursor 12 is in the second position P2. During the process of the handle 20 moving from the first position P1 to the third position P3, the inertial measurement module 200 can measure the inertial data and transmit the inertial data to the processor 250 of the handle 20, and the calibration module 210 can measure the calibration data and transmit the calibration data to the processor 250 of the handle 20.

[0080] Then, the processor 250 of the handle 20 transmits the inertial data and calibration data to the processor 101 of the XR headset 10. After acquiring the inertial data and calibration data, the processor 101 of the XR headset 10 fuses and calculates the inertial data to calibrate the inertial data, thereby obtaining the first displacement data of the handle 20 moving from the first position P1 to the third position P3. It can be understood that, based on the working principle of the aforementioned inertial measurement unit, the first displacement data is data in the rectangular coordinate system O-XYZ, and the first displacement data can be, for example, (ΔX1, ΔY1, ΔZ1).

[0081] Next, the processor 101 of the XR headset 10 converts the first displacement data (ΔX1, ΔY1, ΔZ1) to obtain the second displacement data of the handle 20, which is data in spherical coordinates. For example, Figure 5A and Figure 5B The spherical coordinate system in the embodiments of this application is shown. An exemplary setup method, wherein, Figure 5A It is a 3D image. Figure 5B This is a top view. (Reference) Figure 5A and Figure 5B Corresponding to the user's left and right eyes, the XR head-mounted display device 10 includes a left lens 109L and a right lens 109R. The center M1 of the left lens corresponds to the user's left pupil, and the center M2 of the right lens corresponds to the user's right pupil. The line connecting the centers M1 and M2 is called the first line M1M2. The midpoint M3 of the first line M1M2 is a spherical coordinate system. (As an example of the first spherical coordinate system) The origin O1. In the spherical coordinate system Below, the second displacement data could be, for example, It should be noted that, in this embodiment, the data from the first displacement data (ΔX1, ΔY1, ΔZ1) to the second displacement data... The transformation is simply from the rectangular coordinate system O-XYZ to the spherical coordinate system. The conversion of coordinate representation does not involve scaling up or down displacement data.

[0082] Continue reading Figure 4 The processor 101 of the XR headset 10 obtains the second displacement data of the controller 20. Then, the second displacement data of the handle 20 can be processed according to the preset mapping relationship. Magnify the image to obtain the displacement data of cursor 12. The displacement data of cursor 12 is in spherical coordinates. (As an example of a second spherical coordinate system) the data. Therefore, based on the displacement data of cursor 12. This controls the cursor 10 to move from the second position P2 to the fourth position P4. Thus, only a small movement of the handle 20 is needed to satisfy the large-scale movement requirements of the cursor 12.

[0083] The following formulas (1) to (3) schematically illustrate a second displacement data. Displacement data of cursor 12 The pre-defined mapping relationship between them.

[0084] Δr2=a Δr1 (a>1) (1)

[0085] Δθ2=K1Δθ1(K1>1) (2)

[0086]

[0087] Formula (1) represents a preset exponential mapping relationship between the radial data Δr1 in the second displacement data of the handle 20 and the radial data Δr1 in the displacement data of the cursor 12. Based on this preset exponential mapping relationship, exponentially amplifying the radial data Δr1 in the second displacement data of the handle 20 yields the radial data Δr2 in the displacement data of the cursor 12. Thus, in the radial direction, the greater the movement distance of the handle 20, the more exponentially the movement distance of the cursor 12 increases, thereby better adapting to application scenarios involving interaction from near to far distances.

[0088] Taking a=5 as an example, when the radial data Δr1 of handle 20 is 2cm, the radial data Δr2 of cursor 12 is 25cm. For example, cursor 12 can be viewed from... Figure 5B In the example shown, concentric circles F1 move to concentric circles F2; ​​when the radial data Δr1 of handle 20 is 3cm, the radial data Δr2 corresponding to cursor 12 is 125cm, and cursor 12 can, for example, move from... Figure 5B In the example shown, the concentric circle F2 moves to the more distant concentric circle F3. During this process, the cursor 12 can move between different concentric circles (e.g., concentric circles F1 to F3) without significantly changing the radial position of the handle 20. Furthermore, when controlling the cursor 12 to move radially to the more distant concentric circle F3, the handle 20 does not need to move a proportionally larger distance radially, thus improving the user's control experience.

[0089] Formula (2) represents a first preset proportional mapping relationship between the polar angle data Δθ1 in the second displacement data of the handle 20 and the polar angle data Δθ2 in the displacement data of the cursor 12. Based on this first preset proportional mapping relationship, the polar angle data Δθ1 in the second displacement data of the handle 20 is proportionally amplified to obtain the polar angle data Δθ2 of the cursor 12. Thus, by moving the handle 20 up and down only slightly, a large vertical movement of the cursor 12 can be achieved.

[0090] Formula (3) is the azimuth data of the second displacement data of handle 20. The azimuth data relative to the displacement data of cursor 12 A second preset proportional mapping relationship exists between them. Based on this second preset proportional mapping relationship, the azimuth data in the second displacement data of the handle 20 is... By scaling the image, we can obtain the azimuth data for cursor 12. Thus, by simply moving the handle 20 slightly left and right, the cursor 12 can be moved significantly left and right.

[0091] Wherein, K1 and K2 may be equal or unequal, that is, the first preset ratio mapping relationship and the second preset ratio mapping relationship may be the same or different, and this application does not impose any restrictions on this.

[0092] In summary, by amplifying the displacement data through the preset mapping relationship shown in the above formulas (1) to (3), the handle 20 can drive the cursor 12 to move over a large range when moving within a small range, making the operation simpler and more convenient.

[0093] It is understood that the above formulas (1) to (3) only schematically illustrate some of the preset mapping relationships and do not constitute a limitation on the implementation of this application.

[0094] It is also understood that the above-described scheme of XR head-mounted display device 10 acquiring displacement data of handle 20 during movement and determining the position of cursor 12 based on the displacement data is merely an illustrative example, and those skilled in the art can make other modifications.

[0095] For example, in this embodiment, the processor 101 of the XR headset 10 processes inertial data, calibration data, and displacement data of the handle 20, etc., and the processor 250 of the handle 20 does not need to perform complex data processing. Therefore, the processor 101 of the handle 20 does not need to have high processing performance, thereby effectively reducing the cost of the handle 20, but this application is not limited to this. In some other embodiments, the processor 101 of the handle 20 can also process the inertial data and calibration data to obtain the displacement data of the handle 20, and then transmit the displacement data of the handle 20 to the processor 101 of the XR headset 10.

[0096] For example, in this embodiment, the first displacement data (ΔX1, ΔY1, ΔZ1) of the handle 20 is first converted into spherical coordinates. The second displacement data below Then, the second displacement data... The radial data Δr1 is exponentially amplified, and the polar angle data Δθ2 and azimuth angle data are also amplified. By scaling up, the displacement data of cursor 12 is obtained. However, this application is not limited thereto. In some other embodiments, the second displacement data may also be used. Radial data Δr1, polar angle data Δθ2, azimuth data The displacement data of cursor 12 were obtained by scaling up or exponentially scaling up the data. In other embodiments, the first displacement data (ΔX1, ΔY1, ΔZ1) of the handle 20 can be amplified (e.g., proportionally or exponentially) to obtain the displacement data (ΔA1, ΔB1, ΔC1) of the cursor 12 in the Cartesian coordinate system O1-ABC, without having to convert the displacement data to data in the spherical coordinate system.

[0097] For example, in this embodiment, the second displacement data of the handle 20 In spherical coordinate system The data below, in the same spherical coordinate system Below, the second displacement data of handle 20 Magnify the image to obtain the displacement data of cursor 12. However, this application is not limited thereto. In other embodiments, the second displacement data of the handle 20 It can also be data in other spherical coordinate systems, such as a spherical coordinate system with the same origin and three axes as the Cartesian coordinate system O-XYZ (as another example of the first spherical coordinate system). That is to say, the spherical coordinate system corresponding to the second displacement data of handle 20 can be different from the spherical coordinate system corresponding to the displacement data of cursor 12.

[0098] For example, in this embodiment, the displacement data of the cursor 12 is obtained by magnifying the displacement data of the handle 20, but this application is not limited to this. In other embodiments, the displacement data of the cursor 12 can also be obtained by reducing the displacement data of the handle 20 (e.g., proportional reduction, exponential reduction, etc.) to meet the usage requirements of different application scenarios. The specific implementation method can refer to the description of obtaining the displacement data of the cursor 12 by magnifying the displacement data of the handle 20 in the above embodiments, which will not be repeated here. Thus, in some scenarios that require precise operation (e.g., graphic design), by reducing the displacement data of the handle 20, the corresponding displacement data of the cursor 12 can be made smaller, thereby allowing for more precise control of the movement of the cursor 12. Alternatively, when the movable area of ​​the cursor 12 is limited, obtaining the displacement data of the cursor 12 by reducing the displacement data of the handle 20 can ensure that the cursor 12 can move effectively within the movable area.

[0099] Continue reading Figure 3To further improve the measurement accuracy of the inertial data of the handle 20, in some embodiments of this application, the inertial measurement module 200 of the handle 20 may include a first inertial measurement unit 201 and a second inertial measurement unit 202. Both the first inertial measurement unit 201 and the second inertial measurement unit 202 can measure the inertial data of the handle 20 during movement. By combining the inertial data measured by the first inertial measurement unit 201 and the second inertial measurement unit 202, the measurement error can be further reduced, thereby improving the overall measurement accuracy.

[0100] However, this application is not limited to this. In other embodiments, the inertial measurement module 200 may also include more or fewer inertial measurement units, such as one, three or four.

[0101] In some embodiments of this application, the calibration module 210 of the handle 20 may include an electronic compass. The electronic compass is used to measure the geomagnetic declination data of the handle 20 as it moves. The geomagnetic declination data can provide azimuth information, effectively correct for errors accumulated during IMU integration, and is relatively inexpensive.

[0102] In some embodiments of this application, the calibration module 210 of the handle 20 may include a global positioning system (GPS). GPS is used to measure the latitude and longitude data of the handle 20 as it moves, thereby effectively correcting the errors accumulated during the IMU integration process.

[0103] It should be noted that the above embodiments only describe in detail the working principles of the inertial measurement module 200 and calibration module 210 in the handle 20, and do not constitute a limitation on the implementation of this application. In other embodiments, the handle 20 may also include more devices, such as a power module 220, an input module 230, a communication module 240, and a processor 250, which will be described in detail below.

[0104] Continue reading Figure 3 In some embodiments of this application, the power module 220 may include a battery 221 and a power integrated circuit (IC) 222. The battery 221 provides power to the controller 20, and the battery IC 222 converts the power provided by the battery 221 into a voltage suitable for the various devices in the controller 20. For example, the battery 221 may provide a voltage of 1.5V, and the battery IC 222 may convert the 1.5V voltage into a voltage of 3.3V for use by the processor 250.

[0105] In some embodiments of this application, the input module 230 may include various buttons, such as a return-to-origin button 231, a three-dimensional (3D) / two-dimensional (2D) switching button 232, a confirmation button 233, an auxiliary button 234, a scroll wheel 235, etc. Users can input different commands by clicking different buttons, and then interact with the virtual scene constructed by the XR head-mounted display device 10 in different ways. The specific functions of each button are described in turn below with reference to the accompanying drawings.

[0106] Among them, reference Figure 3 and combined Figure 1 The origin return button 231 is used to control the cursor 12 in the display interface 11 to return to the cursor origin M in the display interface 11. In this way, no matter where the cursor 12 is, clicking the origin return button 231 will make the cursor 12 return to the cursor origin M, thereby achieving quick positioning, avoiding the cursor 12 being lost, and helping to improve the convenience of operation.

[0107] Furthermore, when cursor 12 moves abnormally, it can be returned to the cursor origin M by clicking the origin return button 231 for recalibration, allowing cursor 12 to move normally again. Specifically, the inertial measurement module 200 and calibration module 210 can use the position of the handle 20 corresponding to cursor 12 returning to the cursor origin M as the initial position to restart the measurement of the inertial data and calibration data of handle 20 during movement. In this way, the accumulated errors of the inertial measurement module 200 and calibration module 210 during the measurement process before clicking the origin return button 231, such as errors caused by temperature drift and zero bias, can be effectively eliminated, thereby accurately obtaining the displacement data of handle 20 and enabling cursor 12 to move normally.

[0108] In this embodiment, to improve the user's visual experience, it is necessary to reasonably set the position of the cursor origin M in the display interface 11. An exemplary method for setting the cursor origin M is described below with reference to the accompanying drawings.

[0109] Continue to refer to Figure 5B In some embodiments of this application, the cursor origin M can be located at the user's visual center. Specifically, the line connecting the midpoint M3 of the first line M1M2 and the cursor origin M is the second line M3M. The second line M3M is perpendicular to the first line M1M2 and parallel to the optical axis L1 of the left lens 109L. In this way, the user can see the cursor origin M by looking straight ahead without having to look up, down, or to the left or right, thereby improving the user's visual experience.

[0110] In some implementations, the length of the second connection M3M can be 40cm to 120cm, for example, 40cm, 41cm, 42cm or 43cm, so that users can more easily see the cursor origin M.

[0111] It is understandable that the above Figure 5B This illustration merely demonstrates one way of setting the cursor origin M and does not constitute a limitation of this application. In other embodiments, users can also customize the position of the cursor origin M according to their personal preferences, making the origin return button 231 more in line with their personal usage habits. For example, for nearsighted users, the cursor origin M can be set closer to the user's eyes so that the user can see the cursor origin M clearly.

[0112] Continue reading Figure 3 The 3D / 2D switching button 232 in the input module 230 is used to control the movement mode of the cursor 12 between spatial movement mode and planar movement mode to adapt to different application scenarios and meet customized usage habits. For example, in a game scene, the 3D / 2D switching button 232 can be used to switch the movement mode of the cursor 12 to spatial movement mode. At this time, the cursor 12 has three degrees of freedom (3DOF) and can move in three directions (e.g., ...). Figure 5A The cursor 12 moves along the A-axis, B-axis, and C-axis directions in the illustrated embodiment. This allows the user to interact with any object in the game scene using the cursor 12, providing an immersive gaming experience. For example, in a web browsing scenario, the 3D / 2D switch button 232 can be used to switch the cursor 12's movement mode to planar movement mode. In this mode, the cursor 12 has two degrees of freedom and can move in two directions (e.g., ...). Figure 5A The cursor moves along the B-axis and C-axis directions in the illustrated embodiment, allowing the user to browse web pages using the cursor 12.

[0113] The confirmation button 233 in the input module 230 is similar to the left mouse button, used to confirm content after the cursor 12 moves to a certain position. For example... Figure 1 In the example shown, after the cursor 12 moves to the location of object 13, the selection of object 13 can be confirmed by clicking the confirmation button 233.

[0114] The auxiliary button 234 in the input module 230 is similar to the right mouse button, providing auxiliary functions after selection by the cursor 12. For example... Figure 1 In the example shown, after the cursor 12 selects object 13, the user can click the auxiliary button 234 to perform operations such as rotating and scaling on object 13.

[0115] The scroll wheel 235 in the input module 230, similar to a mouse wheel, is used to control webpage scrolling for user browsing. In some implementations, the handle 20 may include a scroll wheel encoder 260. When the scroll wheel 235 begins to scroll, the scroll wheel encoder 260 detects the scrolling direction and speed of the scroll wheel 235 and generates a corresponding electrical signal. The processor 250 can acquire this electrical signal and scroll the webpage accordingly.

[0116] After introducing the various buttons included in the input module 230, the following section describes an exemplary configuration of multiple buttons in conjunction with the accompanying drawings.

[0117] In some embodiments of this application, at least some of the buttons are distributed in a five-finger shape to better match the position of the user's fingers, resulting in better grip comfort and easier operation.

[0118] Figure 6 This invention illustrates an exemplary distribution of multiple buttons in the handle 20 according to an embodiment of the present application. (Refer to...) Figure 6 In some implementations, the origin return button 231 can correspond to the user's thumb position, the confirmation button 233 can correspond to the user's index finger position, the auxiliary button 234 can correspond to the user's middle finger position, and the 3D / 2D button 232 can correspond to the user's ring finger position. A scroll wheel 235 is positioned between the confirmation button 233 and the auxiliary button 234.

[0119] When using the controller 20, the user can place their palm on it, with their thumb on the home return button 231, index finger on the confirmation button 233, middle finger on the auxiliary button 234, and ring finger on the 3D / 2D button 232. This allows the user to control the home return button 231 with their thumb, the confirmation button 233 with their index finger, the auxiliary button 234 with their middle finger, and the 3D / 2D button 232 with their ring finger. When the scroll wheel 235 needs to be used, the index finger can be moved to the scroll wheel 235 to push it to roll. The overall operation is simple and convenient, and the hand grip is comfortable.

[0120] It is understandable that the above Figure 6 This only shows the distribution of some buttons in the handle 20 and does not constitute a specific limitation on the implementation of this application.

[0121] In some embodiments of this application, the buttons in the input module 230 described above can be physical buttons, that is, buttons with a physical form. Physical buttons can provide users with good tactile feedback, making the user experience better.

[0122] However, this application is not limited to this. In other embodiments of this application, the buttons in the input module 230 described above can also be virtual buttons, that is, buttons displayed on a touch screen or other interface that have no physical form. Virtual buttons are more flexible and their functions or positions can be flexibly configured according to different usage needs.

[0123] Continue reading Figure 3 In some embodiments of this application, the communication module 240 is used to enable communication between the handle 20 and the XR head-mounted display device 10.

[0124] In some embodiments of this application, the processor 250 is used to control the overall operation of the handle 20. For example, the processor 250 can receive inertial data from the inertial measurement module 200, calibration data from the calibration module 210, and electrical signals from the roller encoder 260. As another example, the processor 250 can communicate with the XR head-mounted display device 10 via the communication module 240 to transmit data (e.g., inertial data, calibration data, etc.) to the XR head-mounted display device 10.

[0125] In some implementations, the processor 250 and the inertial measurement module 200 can communicate via an inter-integrated circuit (I2C) or a serial peripheral interface (SPI) to achieve data transmission.

[0126] In some implementations, the processor 250 can communicate with the calibration module 210 and the roller encoder 260 via I2C to achieve data transmission.

[0127] In some of these implementations, the processor 250 can be a microcontroller unit.

[0128] This application also provides an interaction method applicable to the aforementioned XR head-mounted display device 10. It is understood that the XR head-mounted display device 10 is the subject executing this interaction method, and this will not be elaborated further below.

[0129] Figure 7 A flowchart of an interaction method according to an embodiment of this application is shown. (Reference) Figure 7 The interaction method provided in this application specifically includes the following steps:

[0130] S110: Acquire the inertial data and calibration data of the handle 20 when it moves, and calibrate the inertial data according to the calibration data to obtain the displacement data of the handle 20 when it moves.

[0131] refer to Figure 4 and combined Figure 1In some embodiments of this application, when the handle 20 moves from the first position P1 to the third position P3, the inertial measurement module 200 of the handle 20 can measure inertial data, and the calibration module 210 can measure calibration data. The XR head-mounted display device 10 can acquire the inertial data and calibration data, and calibrate the inertial data according to the calibration data to obtain the first displacement data (ΔX1, ΔY1, ΔZ1) when the handle 20 moves.

[0132] In some implementations, calibration data and inertial data can be fused and calculated using fusion algorithms such as Kalman filters, complementary filters, and particle filters to achieve calibration of the inertial data. This application does not impose specific limitations on this.

[0133] S120: Convert the displacement data of the handle 20 when it moves into the displacement data of the cursor 12.

[0134] According to the above Figure 4 and Figure 1 In the illustrated embodiment, after acquiring the first displacement data (ΔX1, ΔY1, ΔZ1) when the controller 20 moves, the XR head-mounted display device 10 can convert the first displacement data (ΔX1, ΔY1, ΔZ1) into second displacement data when the controller 20 moves. Then, the second displacement data is mapped according to the preset exponential mapping relationship. The radial data Δr1 in the image is exponentially amplified to obtain the radial data Δr2 of cursor 12; the second displacement data is then mapped according to the first preset proportional relationship. The polar angle data Δθ1 in the image is scaled up to obtain the polar angle data Δθ2 of cursor 12; the second displacement data is then mapped according to the second preset scaling relationship. Azimuth data in By scaling up, we can obtain the azimuth data for cursor 12. Finally, the displacement data of cursor 12 is obtained.

[0135] Among them, the second displacement data is based on a preset mapping relationship. Magnify the image to obtain the displacement data of cursor 12. For specific implementation methods and beneficial effects, please refer to the above. Figure 4 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0136] In this way, only a small range of movement of the handle 20 is needed to meet the large range of movement requirements of the cursor 12, without having to raise the handle 20 to aim at objects in the virtual scene constructed by the XR head-mounted display device 10, making the operation simple and convenient.

[0137] It is understandable that there are many ways to convert the displacement data of the handle 20 when it moves into the displacement data of the cursor 12, and these methods are not limited to those described above. Figure 4 The conversion method in the illustrated embodiment.

[0138] For example, in some other embodiments, the second displacement data can also be... Radial data Δr1, polar angle data Δθ2, azimuth data The displacement data of cursor 12 were obtained by scaling up or exponentially scaling up the data.

[0139] In other embodiments, the first displacement data (ΔX1, ΔY1, ΔZ1) of the handle 20 can be amplified (e.g., proportionally or exponentially) to obtain the displacement data (ΔA1, ΔB1, ΔC1) of the cursor 12 in the Cartesian coordinate system O1-ABC, without having to convert the displacement data to data in the spherical coordinate system.

[0140] In some other embodiments, the second displacement data of the handle 20 It can also be data in other spherical coordinate systems, such as a spherical coordinate system with the same origin and three axes as the Cartesian coordinate system O-XYZ (as another example of the first spherical coordinate system). That is to say, the spherical coordinate system corresponding to the second displacement data of handle 20 can be different from the spherical coordinate system corresponding to the displacement data of cursor 12.

[0141] In other embodiments, the displacement data of the cursor 12 can also be obtained by scaling down the displacement data of the handle 20 (e.g., proportional scaling down, exponential scaling down, etc.) to meet the usage requirements of different application scenarios. Thus, in scenarios requiring precise operation (e.g., graphic design), scaling down the displacement data of the handle 20 results in smaller displacement data for the corresponding cursor 12, allowing for more precise control over the movement of the cursor 12. Alternatively, when the movable area of ​​the cursor 12 is limited, scaling down the displacement data of the handle 20 to obtain the cursor 12's displacement data ensures that the cursor 12 can move effectively within the movable area.

[0142] S130: Determine the position of cursor 12 based on the displacement data of cursor 12.

[0143] According to the above Figure 4 and Figure 1 In the embodiment shown, the displacement data of cursor 12 is obtained. Then, based on the displacement data of cursor 12 The required movement distance of cursor 12 is determined, so that cursor 12 can be moved from the second position P2 to the fourth position P4 to select rectangular object 13 and interact with rectangular object 13, such as zooming in, zooming out or dragging rectangular object 13.

[0144] In the above interaction method, the XR headset 10 determines the position of the cursor 10 by acquiring the displacement data of the handle 20 during movement, rather than acquiring the position of the handle 20 relative to the XR headset 10. This eliminates the need for a dedicated camera module to capture images from the handle 20, effectively reducing costs. Furthermore, the handle 20 does not require additional optical components such as infrared or visible light sources, facilitating its miniaturization. Simultaneously, the integrated positioning method combining the inertial measurement module 200 and the calibration module 210 helps accurately identify the displacement data of the handle 20 during movement, resulting in high detection accuracy.

[0145] It is understandable that the above Figure 7 This application merely illustrates one interaction method applied in an XR headset 10, and is not limited thereto. For example, in some other embodiments, the processor 101 of the handle 20 may calibrate the inertial data based on calibration data to obtain the displacement data of the handle 20, and transmit this displacement data to the XR headset 10. The XR headset 10 then determines the position of the cursor 10 based on the displacement data of the handle 20. In other words, the XR headset 10 acquires the displacement data of the handle 20, rather than the inertial data and calibration data.

[0146] This application also provides an XR head-mounted display device 10 for performing the above-described interaction method. Exemplarily, Figure 8 An exemplary structural block diagram of an XR head-mounted display device 10 in an embodiment of this application is shown.

[0147] like Figure 8 As shown, the XR head-mounted display device 10 may include a processor 101, a memory 102, a sensor module 103, an audio module 104, buttons 105, an input / output interface 106, a communication module 107, a power supply module 108, and a display screen 109, etc.

[0148] Processor 101 is typically used to control the overall operation of XR headset device 10 and may include one or more processing units. For example, processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0149] The processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. This memory can store instructions or data that the processor 101 has just used or is recurring. If the processor 101 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system. In this application, the processor 101 can execute the interactive methods mentioned in this application, such as those described above. Figure 7 The interaction method in the illustrated embodiment.

[0150] In some embodiments, the processor 101 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0151] The memory 102 can be used to store computer executable program code, which includes instructions. The processor 101 executes various functional applications and data processing of the XR headset 10 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (e.g., image display function, video playback function), etc. The data storage area may store data generated during the use of the XR headset 10 (e.g., the first displacement data (ΔX1, ΔY1, ΔZ1) and second displacement data of the handle 20 calculated by the XR headset 10). displacement data of cursor 12 In addition, memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0152] The sensor module 103 may include sensors for detecting the working and usage status of the XR headset 10. These include, but are not limited to, optical sensors / pressure sensors for detecting whether the user is wearing the XR headset 10, and inertial sensors for detecting the user's head movement.

[0153] The audio module 104 may include a speaker, a microphone, etc., for implementing audio functions.

[0154] There may be one or more buttons 105. The form of buttons 105 may be a button, a switch, a dial, or a touch or proximity touch sensing device (such as a touch sensor). The XR head-mounted display device 10 can trigger corresponding functions based on the user's operation of one or more of the buttons 105, such as adjusting the position of the first lens barrel 110 and the second lens barrel 120, playing audio, etc.

[0155] The input / output interface 106 can connect other devices to the XR headset 10 via suitable components. Components may include, for example, audio / video jacks, data connectors, etc.

[0156] The communication module 107 may include a wireless communication module. Wireless communication functionality can be implemented using an antenna (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The XR headset 10 may include multiple antennas, each capable of covering one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antennas can be multiplexed as diversity antennas for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0157] The wireless communication module can provide solutions for wireless communication applications on the XR headset 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 101. The wireless communication module can also receive signals to be transmitted from the processor 101, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0158] The power supply module 108 may include a battery for supplying power to the various modules of the XR head-mounted display device 10.

[0159] The display lens 109 may include a left lens 109L and a left lens 109R. The left lens 109L is used to display an image to the user's left pupil, and the left lens 109R is used to display an image to the user's right pupil.

[0160] Understandable. Figure 8 The structure of the XR headset 10 shown is only an example. In other embodiments, the XR headset 10 may include more or fewer modules, or some modules may be combined or split, which is not limited here.

[0161] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described actions. Figure 7 The adjustment method in the illustrated embodiment.

[0162] The specific embodiments described above illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with some embodiments, this does not mean that the features of this application are limited to this embodiment. On the contrary, the purpose of describing the application in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of this application. This application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details have been omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0163] In the description of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "outer", "inner", "circumferential", "radial", "axial", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0164] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "set," "install," "connect," and "fit" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0165] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An interactive system, characterized by The head-mounted device and the handle connected by communication connection, wherein: The head-mounted device is configured to display a display interface; The handle comprises an inertial measurement module and a calibration module, the inertial measurement module is configured to measure inertial data when the handle moves, and the calibration module is configured to measure calibration data when the handle moves, the calibration data being position or angle data of the handle when moving with the earth as a reference system; The head-mounted device is capable of acquiring displacement data of the handle when moving, and determining a position of a cursor in the display interface according to the displacement data of the handle, wherein the displacement data of the handle is obtained by calibrating the inertial data according to the calibration data.

2. The interactive system of claim 1, wherein, The calibration module comprises an electronic compass, and the calibration data comprises magnetic declination data.

3. The interactive system of claim 1, wherein, The inertial measurement module comprises a plurality of inertial measurement units, and the inertial data comprises inertial data measured by the plurality of inertial measurement units.

4. The interactive system of claim 1, wherein, The head-mounted device is capable of acquiring the inertial data and the calibration data, and calibrating the inertial data according to the calibration data to obtain displacement data of the handle when moving.

5. The interactive system of claim 1, wherein, The head-mounted device determines the position of the cursor in the display interface according to the displacement data of the handle, comprising: The head-mounted device converts the displacement data of the handle into displacement data of the cursor; The head-mounted device determines the position of the cursor in the display interface according to the displacement data of the cursor.

6. The interactive system of claim 5, wherein, The displacement data of the handle is obtained by converting the displacement data of the handle from a rectangular coordinate system into a first spherical coordinate system; The head-mounted device converts the displacement data of the handle into displacement data of the cursor, comprising: The head-mounted device magnifies radial data in the displacement data of the handle according to a preset exponential mapping relationship to obtain radial data of the cursor, magnifies azimuth data in the displacement data of the handle according to a first preset proportional mapping relationship to obtain azimuth data of the cursor, magnifies polar angle data in the displacement data of the handle according to a second preset proportional mapping relationship to obtain polar angle data of the cursor, and takes the radial data, the polar angle data and the radial data of the cursor as the displacement data of the cursor, the displacement data of the cursor being data in a second spherical coordinate system.

7. The interactive system of claim 6, wherein, The head-mounted device comprises left and right eye lenses, a first connecting line between the centers of the left and right eye lenses, and a midpoint of the first connecting line as an origin of the first spherical coordinate system.

8. An interaction system according to claim 6 or 7, characterised in that, The first spherical coordinate system is the same as the second spherical coordinate system.

9. The interactive system of claim 1, wherein, The handle comprises an origin regression button configured to control the cursor in the display interface to return to a cursor origin in the display interface.

10. The interactive system of claim 9, wherein, The head-mounted device comprises left and right eye lenses, a first connecting line between the centers of the left and right eye lenses; A second connecting line between the cursor origin and the midpoint of the first connecting line is perpendicular to the first connecting line and parallel to the optical axis direction of the left eye lens.

11. The interactive system of claim 10, wherein, The length of the second connecting line is 40cm-120cm.

12. The interactive system of claim 1, wherein, The handle comprises a 3D / 2D switching key for controlling the cursor movement mode to switch between a space movement mode and a plane movement mode.

13. The interactive system of claim 1, wherein, The handle comprises a plurality of keys arranged in an extended five-finger shape.

14. An interaction method, characterized in that, The head-mounted device in communication connection with the handle is used to display a display interface, and the method comprises: Obtaining inertia data and calibration data when the handle moves, and calibrating the inertia data according to the calibration data to obtain displacement data when the handle moves, wherein the calibration data is position or angle data of the handle moving with the earth as a reference system; Converting the displacement data of the handle moving into displacement data of the cursor in the display interface; Determining the position of the cursor in the display interface according to the displacement data of the cursor.

15. The interaction method of claim 14, wherein, The calibration data comprises geomagnetic declination data.

16. The interaction method of claim 14, wherein, The displacement data of the handle is converted from a rectangular coordinate system into a first spherical coordinate system; Converting the displacement data of the handle into the displacement data of the cursor comprises: According to a preset index mapping relationship, the radial data in the displacement data of the handle is amplified to obtain the radial data of the cursor, according to a first preset proportion mapping relationship, the azimuth data in the displacement data of the handle is amplified to obtain the azimuth data of the cursor, according to a second preset proportion mapping relationship, the polar angle data in the displacement data of the handle is amplified to obtain the polar angle data of the cursor, and the radial data, the polar angle data and the radial data of the cursor are taken as the displacement data of the cursor, the displacement data of the cursor being data under a second spherical coordinate system.

17. The interaction method of claim 16, wherein, The head-mounted device comprises left and right eye lenses, the line connecting the centers of the left and right eye lenses is a first line, and the midpoint of the first line is the origin of the first spherical coordinate system.

18. The interaction method according to claim 16 or 17, characterized in that, The first spherical coordinate system is the same as the second spherical coordinate system.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the interaction method of any one of claims 14 to 18.

20. A head-mounted device, comprising: Comprise: a memory for storing instructions, and one or more processors, when the instructions are executed by the one or more processors, the processor executes the interaction method as claimed in any one of claims 14 to 18.

Images