Method and system for determining relative posture relationship between at least two users

Abstract

The present application relates to a method and an interactive system for determining a relative position and posture relationship between at least two users, the method comprising: marking, by a controller, the spatial position of a second user wearing a second head-mounted display device in the field of view of a first user wearing a first head-mounted display device; marking, by the controller, the spatial position of an object or a part of an object aimed at by the second user through a crosshair on a display screen of the second head-mounted display device in the field of view of the first user wearing the first head-mounted display device; and determining, based on the spatial position of the second user wearing the second head-mounted display device and the spatial position of the object or a part of the object aimed at by the crosshair, the relative position and posture relationship between the first user wearing the first head-mounted display device and the second user wearing the second head-mounted display device.

Description

Technical Field

[0001] The present invention relates to the field of information interaction, and in particular to a method and system for determining a relative posture relationship between at least two users. Background Art

[0002] The statements in this section are merely intended to provide background information related to the technical solution of the present application to aid understanding, and they do not necessarily constitute prior art with respect to the technical solution of the present application.

[0003] In recent years, augmented reality (AR) and virtual reality (VR) technologies have gained increasing popularity. In AR / VR applications, the commonly used display and interaction devices are handheld smart devices (such as mobile phones) and head-mounted display devices (such as AR / VR glasses, smart glasses, and AR / VR helmets). Virtual content to be displayed to the user can be presented on the display screen of a mobile phone or head-mounted display device, and the user can interact with this virtual content. Examples of virtual content include icons, images, text, emoticons, virtual people, virtual three-dimensional objects, three-dimensional models, animations, videos, and so on.

[0004] In AR / VR applications, users of head-mounted display devices need to determine their position and / or posture information (hereinafter collectively referred to as "pose information"), and use this position and / or posture information to determine which virtual content to present and how to present it. In some cases, two or more users may need to experience AR / VR applications together, such as multiplayer gaming, multiplayer social networking, multiplayer teaching, doctor-patient communication, etc., which requires binding multiple head-mounted display devices or multiple head-mounted display device users to a single coordinate system. One traditional positioning and posture determination method is to pre-build a three-dimensional point cloud for the scene and determine the position and posture information of the head-mounted display device in the scene based on the three-dimensional point cloud. Another traditional positioning and posture determination method is to pre-deploy one or more positioning base stations or positioning markers in the scene and use the positioning base stations or positioning markers to determine the position and posture information of the head-mounted display device in the scene. However, these methods are relatively cumbersome and costly, requiring advance preparation or deployment, such as pre-building a three-dimensional point cloud for the scene or pre-deploying positioning base stations or positioning markers. Moreover, these methods lack flexibility and cannot be implemented anytime and anywhere. Summary of the Invention

[0005] One aspect of the present invention relates to a method for determining a relative positional relationship between at least two users, wherein a first user wears a first head-mounted display device and has a controller configured for the first head-mounted display device, and a second user wears a second head-mounted display device, and one or more displays of the second head-mounted display device can present a crosshair, and the crosshair can be used to align with any position in space. The method includes: marking, by the controller, the spatial position of the second user wearing the second head-mounted display device in the field of view of the first user wearing the first head-mounted display device; marking, by the controller, the spatial position of the object or a part of the object that the second user is aligning with through the crosshair on the display of the second head-mounted display device in the field of view of the first user wearing the first head-mounted display device; and determining the relative position and posture relationship between the first user wearing the first head-mounted display device and the second user wearing the second head-mounted display device based on the spatial position of the second user wearing the second head-mounted display device and the spatial position of the object or a part of the object aligned with the crosshair.

[0006] One aspect of the present invention relates to a storage medium storing a computer program. When the computer program is executed by a processor, the computer program can be used to implement the method described in the present application.

[0007] One aspect of the present invention relates to an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it can be used to implement the method described in the present application.

[0008] One aspect of the present invention relates to an interactive system, including a controller and a head-mounted display device, wherein the controller and the head-mounted display device can be used to implement the method described in this application.

[0009] The solution of the present application can quickly determine the relative posture relationship between multiple users wearing head-mounted display devices without relying on scene three-dimensional point clouds, positioning base stations, positioning markers, etc., so it has extremely low implementation costs. In addition, it can be implemented anytime and anywhere, with great convenience and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0011] Figure 1 An interactive system based on a handheld smart device (such as a mobile phone) and a head-mounted display device according to one embodiment is shown;

[0012] Figure 2 An interactive system based on a handheld smart device (such as a mobile phone) and a head-mounted display device according to one embodiment is shown;

[0013] Figure 3 An interactive system based on a handheld smart device (such as a mobile phone) and a head-mounted display device according to one embodiment is shown;

[0014] Figure 4 A method for determining a relative posture relationship between at least two users according to one embodiment is shown;

[0015] Figure 5 A method for determining a relative posture relationship between at least two users according to an embodiment is shown.

[0016] Figure 6 A method for determining a relative posture relationship between at least two users according to one embodiment is shown;

[0017] Figure 7 A method for determining a relative posture relationship between at least two users according to one embodiment is shown; and

[0018] Figure 8 A diagram showing an actual effect according to an embodiment is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Figure 1 The present invention illustrates an interactive system based on a handheld smart device (e.g., a mobile phone) and a head-mounted display device according to one embodiment. The interactive system includes a mobile phone 102 of a user 101 and a head-mounted display device (e.g., AR / VR glasses) 103. User 101 is located in a scene, holding mobile phone 102 in hand and wearing head-mounted display device 103 on their head. They can experience various augmented reality or virtual reality applications through mobile phone 102 and head-mounted display device 103. In this system, mobile phone 102 can serve as a controller for head-mounted display device 103 to implement interactive functions in various augmented reality or virtual reality applications, such as selecting virtual objects, manipulating virtual objects, inputting information, etc. Head-mounted display device 103 has a display screen for presenting various virtual objects in augmented reality or virtual reality applications to user 101.

[0021] The position and / or attitude changes of the mobile phone 102 and the head-mounted display device 103 can be measured or tracked by their respective sensors (e.g., accelerometer, magnetic sensor, orientation sensor, gravity sensor, gyroscope, camera, etc.) using methods known in the art (e.g., inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.), thereby determining the real-time position and / or attitude of the mobile phone and the head-mounted display device in space. In one embodiment, one or more of the mobile phone 102 and the head-mounted display device 103 have 6 degrees of freedom (6DoF) position and attitude tracking capabilities.

[0022] In one embodiment, various virtual objects in augmented reality or virtual reality applications can also be presented on the mobile phone 102 (e.g., on the display screen of the mobile phone 102). In one embodiment, the user 101 can also use the head-mounted display device 103 to implement information interaction functions in augmented reality or virtual reality applications, such as through voice, buttons, touch bar, gestures, eyes, etc.

[0023] In this system, the mobile phone 102 is mainly used for interactive operations, and the head-mounted display device 103 is mainly used for information presentation, thereby making full use of their respective advantages, and the two work together to achieve convenient information display and interactive functions.

[0024] In some cases, two or more users may need to experience AR / VR applications together, such as multiplayer games, multiplayer social networking, multiplayer teaching, doctor-patient communication, etc., in which each participating user wears a head-mounted display device to enable interaction between different users. For example, in multiplayer AR / VR games, one user must be able to observe the virtual bullets fired by other users; in multiplayer AR / VR social networking, one user must be able to observe the virtual tags set by other users for themselves or the messages or graffiti of other users at specific locations in the space; in multiplayer AR / VR teaching, students must be able to synchronously observe the teacher's operations on virtual three-dimensional objects in the space; in AR / VR doctor-patient communication, patients must be able to see the three-dimensional tissue model displayed by the doctor and the various operations performed by the doctor.

[0025] Figure 2An interactive system based on a handheld smart device (e.g., a mobile phone) and a head-mounted display device according to one embodiment is shown. The interactive system includes a mobile phone 102 and a head-mounted display device 103 of user 101, and a head-mounted display device 105 of user 104. User 101 can be, for example, a doctor, and user 104 can be, for example, a patient. User 101 can place a three-dimensional tissue model of user 104 (e.g., a three-dimensional skull model) in space and view it through the head-mounted display device 103. At the same time, user 101 can use the mobile phone 102 to perform various operations on the three-dimensional tissue model, such as moving, scaling, rotating, labeling, and indicating. User 104 can observe the three-dimensional tissue model placed by the doctor at the corresponding position in space through the head-mounted display device she wears, and can observe the various operations performed by the doctor on the three-dimensional tissue model in real time.

[0026] To achieve the aforementioned multi-person AR / VR interaction, it is necessary to bind the head-mounted display devices worn by multiple users to the same spatial coordinate system. This means that it is necessary to be able to determine the relative position and posture relationships between multiple head-mounted display devices or the users of these multiple head-mounted display devices. In this application, position and posture may also be referred to as "pose." Figure 3 The interactive system based on a handheld smart device (such as a mobile phone) and a head-mounted display device according to an embodiment is shown, and some of them are similar to Figure 2 Similarly, no further details will be given here. A crosshair can be displayed on one or more displays of user 104's head-mounted display device 105. User 104 can align the crosshair with any object, part thereof, or any location in space by moving or shaking their head. The crosshair can be any shape, such as a cross, a dot, or a ring, and can be displayed, for example, at the center of the display. In one embodiment, the crosshair can also be displayed at other locations on the display. Figure 3 The system shown further includes an object 108 , which can be any object. The object 108 can be an object located at a fixed position or a movable object, such as a finger of the user 101 .

[0027] When determining the relative position and posture relationship between user 101's head-mounted display device 103 and user 104's head-mounted display device 105, user 104 can rotate their head to align the crosshairs on the display screen of head-mounted display device 105 with a location in space, such as the top of object 108. User 101 can use mobile phone 102 to mark the spatial location 106 of user 104 or their head-mounted display device 105 in the field of view of head-mounted display device 103, as well as the spatial location 107 to which the crosshairs are directed. These two spatial locations are then used to determine the relative position and posture relationship between head-mounted display device 103 and head-mounted display device 105, that is, the relative position and posture relationship between user 101 and user 104, thereby binding the two to a single coordinate system.

[0028] exist Figure 3 , the spatial position 106 of the marked user 104 or its head-mounted display device 105 is shown as the spatial position of the right display screen of the head-mounted display device 105, but it can be understood that it can also be the spatial position of any other component of the head-mounted display device 105, or any other spatial position that can be used to derive or calculate the spatial position of the user 104 or its head-mounted display device 105, such as the position of the left display screen of the head-mounted display device 105, the position of the left eye or right eye of the user 104 of the head-mounted display device 105, the position of the center of the left display screen and the right display screen of the head-mounted display device 105, the position of the center of the head-mounted display device 105, the head position of the user 104, and the like.

[0029] In one embodiment, a crosshair is presented on a display screen of the head mounted display device 105 , and the marked spatial position 106 is the position of the display screen presenting the crosshair, or the position of the user's eyes corresponding to the display screen.

[0030] Figure 4 A method for determining the relative position and posture relationship between at least two users according to one embodiment is shown. The method uses a handheld smart device (e.g., a mobile phone) to determine the relative position and posture relationship between multiple users' head-mounted display devices, thereby determining the relative position and posture relationship between the multiple users. The method includes:

[0031] Step 401: Using the handheld smart device, mark the spatial position of a second user wearing a second head-mounted display device in the field of view of a first user wearing a first head-mounted display device.

[0032] by Figure 3For example, user 101 can use mobile phone 102 (i.e., handheld smart device) to mark the spatial position of user 104's head-mounted display device 105 (i.e., second head-mounted display device) in the field of view of his head-mounted display device 103 (i.e., first head-mounted display device). The spatial position represents the spatial position of user 104. Figure 3 Indicated by 106 .

[0033] The mobile phone 102 can use various feasible methods to mark the spatial position of the user 104 or their head-mounted display device 105 in the field of view of the head-mounted display device 103. In one embodiment, the spatial position of the mobile phone 102 can be used to control the movement of a virtual object (e.g., a cursor, pointer, etc.) in the field of view of the head-mounted display device 103 until it moves to the spatial position of the head-mounted display device 105, that is, the spatial position of the virtual object coincides with the spatial position of the head-mounted display device 105. It should be noted that a virtual object in the field of view of the head-mounted display device 103 has specific three-dimensional coordinates in the spatial coordinate system of the head-mounted display device 103. If the three-dimensional coordinates of the virtual object are changed, its position in the field of view of the head-mounted display device 103 will also change, and vice versa. The user wearing the head-mounted display device 103 can use the mobile phone 102 to control the movement of the virtual object in their field of view so that it coincides with an object or a portion of an object in space. In one embodiment, when determining whether the spatial position of the virtual object coincides with the spatial position of the head-mounted display device 105, the user 101 may appropriately move their body left and right, or move or shake their head left and right, to ensure that the virtual object and the head-mounted display device 105 coincide in three-dimensional space, rather than just in a certain line of sight. This is because if the two coincide only in a certain line of sight, they may have different depths relative to the observer and are not coincident in the sense of spatial position.

[0034] When the virtual object moves to the spatial position of the head-mounted display device 105, the user 101 can send a command (for example, by pressing a button or screen on the mobile phone 102, or by voice) to record the current position of the virtual object, thereby marking the spatial position of the head-mounted display device 105. When marking the spatial position of the head-mounted display device 105, only the position information of the spatial position can be recorded, or a virtual object (for example, a sphere) can be placed at the spatial position to visually present the spatial position to the user 101.

[0035] The recorded spatial position information of the user 104 or the head-mounted display device 105 may be its position information relative to the head-mounted display device 103 of the user 101, for example, the three-dimensional coordinate information of the user 104 or the head-mounted display device 105 in the spatial coordinate system of the head-mounted display device 103.

[0036] Step 402: Mark, by means of the handheld smart device, in the visual field of the first user wearing the first head-mounted display device, the spatial position at which the second user is aiming through the crosshairs on the display screen of the second head-mounted display device.

[0037] by Figure 3 For example, user 101 can use mobile phone 102 to mark the spatial position (that is, the spatial position of the object or a part thereof) that user 104 is aiming at through the crosshairs on the display screen of head-mounted display device 105 in the field of view of head-mounted display device 103, and the spatial position is Figure 3 Indicated by 107.

[0038] The mobile phone 102 can use various feasible methods to mark the spatial position of the crosshairs on the display screen of the head-mounted display device 105 in the field of view of the head-mounted display device 103. In one embodiment, the movement of a virtual object (e.g., a cursor, pointer, etc.) in the field of view of the head-mounted display device 103 can be controlled by changing the spatial position of the mobile phone 102 until it moves to the spatial position of the crosshairs on the display screen of the head-mounted display device 105, that is, the spatial position of the virtual object coincides with the spatial position of the crosshairs. In one embodiment, when determining whether the spatial position of the virtual object coincides with the spatial position of the crosshairs, the user 101 can appropriately move the body left and right, or appropriately move the head left and right, or shake the head to ensure that the two coincide in three-dimensional space, rather than just coinciding in a certain line of sight direction but having different depths.

[0039] When the virtual object moves to the spatial position at which the crosshairs are pointed on the display screen of the head-mounted display device 105, the user 101 can send a command (e.g., by pressing a button or screen on the mobile phone 102, or by voice, etc.) to record the current position of the virtual object, thereby marking the spatial position at which the crosshairs are pointed. When marking the spatial position at which the crosshairs are pointed, only the position information of the spatial position can be recorded, or a virtual object (e.g., a sphere) can be further placed at the spatial position to visually present the spatial position to the user 101.

[0040] The recorded spatial position information of the crosshairs may be position information relative to the head mounted display device 103 of the user 101 , for example, three-dimensional coordinate information of the spatial position of the crosshairs in the spatial coordinate system of the head mounted display device 103 .

[0041] In one embodiment, user 101 and user 104 may agree in advance on the object or a part thereof at which the crosshairs on the display screen of the head-mounted display device 105 are to be aimed, or user 101 may inform user 104 of the object or a part thereof at which the crosshairs are to be aimed, or user 104 may inform user 101 of the object or a part thereof at which the crosshairs are to be aimed.

[0042] Step 403: Determine the relative position and posture relationship between the first user wearing the first head-mounted display device and the second user wearing the second head-mounted display device based on the spatial position of the second user wearing the second head-mounted display device and the spatial position where the crosshairs are aligned.

[0043] After marking the spatial position 106 of the user 104 or its head-mounted display device 105 and the spatial position 107 where the crosshairs are aligned on the display screen of the head-mounted display device 105, the relative position and posture relationship between the user 101 and the user 104 can be determined based on these two spatial positions, which is actually also the relative position and posture relationship between the head-mounted display device 103 and the head-mounted display device 105.

[0044] Specifically, based on the marked spatial position 106 of user 104 or their head-mounted display device 105, the position information of user 104 or their head-mounted display device 105 relative to head-mounted display device 103 can be determined. Based on the marked spatial position 106 of user 104 or their head-mounted display device 105 and the spatial position 107 of the crosshairs on the display screen of head-mounted display device 105, the gaze direction or orientation information of user 104 or their head-mounted display device 105 can be determined. Because users typically keep their eyes roughly aligned when wearing a head-mounted display device for communication (i.e., users typically do not significantly rotate their heads around their gaze), the posture information of user 104 or their head-mounted display device 105 can be determined based on the gaze direction or orientation information of user 104 or their head-mounted display device 105. Consequently, the relative position and posture relationship between user 101 wearing head-mounted display device 103 and user 104 wearing head-mounted display device 105 can be determined.

[0045] In one embodiment, when marking the spatial position of the second user or the second head-mounted display device, or when marking the spatial position at which the second user is aligned with the crosshairs on the display screen of the second head-mounted display device, the mobile phone 102 or other handheld smart device may not be used, but any other form of controller may be used as long as it is capable of marking the spatial position. In one embodiment, the controller may be a controller based on user voice, which marks the spatial position through the user voice. In one embodiment, the controller may be a controller based on user gestures, which marks the spatial position through user gestures. In one embodiment, the controller may be a controller based on the user's line of sight direction, which marks the spatial position through the user's line of sight (i.e., the direction in which the user's gaze is directed). The controller may be connected to the first head-mounted display device by wire or wirelessly, or may be integrated with the first head-mounted display device or be a part of it.

[0046] Figure 5 A method for determining a relative posture relationship between at least two users according to an embodiment is shown, the method comprising (partially with Figure 4 The steps are similar and will not be repeated here):

[0047] Step 501: Using the handheld smart device, mark the spatial position of a second user wearing a second head-mounted display device in the field of view of a first user wearing a first head-mounted display device.

[0048] Step 502: Mark, by means of the handheld smart device, in the visual field of the first user wearing the first head-mounted display device, the spatial position at which the second user is aiming through the crosshairs on the display screen of the second head-mounted display device.

[0049] Step 503: Determine the relative position and posture relationship between the first user wearing the first head-mounted display device and the second user wearing the second head-mounted display device based on the spatial position of the second user wearing the second head-mounted display device and the spatial position where the crosshairs are aligned.

[0050] Step 504: Track position changes and / or posture changes of the first head-mounted display device and the second head-mounted display device to achieve interaction between the first user and the second user.

[0051] After determining the relative position and posture relationship between the user 101 wearing the head-mounted display device 103 and the user 104 wearing the head-mounted display device 105 (i.e., the head-mounted display device 103 and the head-mounted display device 105), the head-mounted display device 103 and the head-mounted display device 105 can respectively measure or track their position changes and / or posture changes through various sensors therein (for example, acceleration sensors, magnetic sensors, orientation sensors, gravity sensors, gyroscopes, cameras, etc.) through methods known in the art (for example, inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.), thereby determining the real-time relative position and / or posture between the head-mounted display device 103 and the head-mounted display device 105, so as to realize interaction between the first user and the second user.

[0052] It should be noted that, depending on different application scenarios or different application requirements, in some embodiments, it is not necessary to perform six degrees of freedom (6DoF) tracking of the position and posture of the first head-mounted display device and / or the second head-mounted display device, but only the position information or only the posture information can be tracked. For example, in one embodiment, considering that there are currently a large number of mid-to-low-end head-mounted display devices that only have 3DOF posture tracking capabilities, the head-mounted display device can only track its posture changes without considering the position changes of the head-mounted display device, or ignore its position changes. Although the position changes of the head-mounted display device are not taken into account, for some augmented reality applications that users usually experience in place or in a specific small area (that is, users usually only need to turn their heads or bodies to experience), this will not have a significant adverse impact on the user experience.

[0053] In one embodiment, the position and / or posture changes of the handheld smart device, the first head-mounted display device, and the second head-mounted display device can be tracked to achieve interaction between them. For example, in multi-person AR / VR teaching, the teacher can use the handheld smart device to perform operations on virtual three-dimensional objects in the field of view of the head-mounted display device, and the students can observe the virtual three-dimensional objects and the operations performed by the teacher in real time and synchronously through their respective head-mounted display devices; in AR / VR doctor-patient communication, the doctor can use the handheld smart device to perform operations on the three-dimensional tissue model in the field of view of the head-mounted display device, and the patient can see the three-dimensional tissue model and the various operations performed by the doctor through his or her head-mounted display device.

[0054] In one embodiment, after a period of use, the relative position and posture between the first head-mounted display device and the second head-mounted display device can be re-determined in the above manner to eliminate possible errors.

[0055] Through the above method, multiple users can wear their own head-mounted display devices to experience augmented reality or virtual reality applications together. At the same time, users can use their mobile phones to perform various operations required in augmented reality or virtual reality applications, such as clicking operation menus, clicking operation buttons, placing virtual objects, deleting virtual objects, aiming at virtual objects, clicking virtual objects, moving virtual objects, rotating virtual objects, drawing virtual works, inputting information, etc.

[0056] In one embodiment, the spatial position of the second head-mounted display device can be marked first, and then the spatial position on the display screen of the second head-mounted display device at which the crosshairs are aligned can be marked. When marking the spatial position at which the crosshairs are aligned, it is not necessarily required that the crosshairs on the display screen of the second head-mounted display device are aligned with that spatial position at that moment. For example, the spatial position of the second head-mounted display device can be marked at a first moment when the crosshairs on the display screen of the second head-mounted display device are aligned with a certain spatial position. The second head-mounted display device can then change its position or posture and, at a second moment, mark the spatial position at which the crosshairs were aligned at the previous moment (i.e., the first moment). This approach requires that the first and second head-mounted display devices be able to track or record their position and posture (posture) change information from the first moment to the second moment. Subsequently, based on the marked spatial position of the second head-mounted display device and the spatial position at which the crosshairs are aligned, the relative position and posture relationship between the first and second head-mounted display devices at the first moment can be determined. Furthermore, based on the position and posture change information from the first moment to the second moment, the relative position and posture relationship between the first and second head-mounted display devices at the second moment can be determined. After the second moment, the position change and / or posture change of the first head-mounted display device and the second head-mounted display device may continue to be tracked to achieve interaction between the two.

[0057] In one embodiment, the spatial position at which the crosshairs are aligned on the display screen of the second head-mounted display device may be marked first, and then the spatial position of the second head-mounted display device may be marked. When marking the spatial position at which the crosshairs are aligned, it is not necessarily required that the crosshairs on the display screen of the second head-mounted display device are aligned with the spatial position at that moment. For example, the spatial position at which the crosshairs are aligned may be marked in advance at a first moment, and then the spatial position of the second head-mounted display device may be marked at a second moment when the crosshairs on the display screen of the second head-mounted display device are aligned with the spatial position. Thereafter, based on the marked spatial position of the second head-mounted display device and the spatial position at which the crosshairs are aligned, the relative position and posture relationship between the first head-mounted display device and the second head-mounted display device at the second moment may be determined. After the second moment, the position changes and / or posture changes of the first head-mounted display device and the second head-mounted display device may continue to be tracked to enable interaction between the two.

[0058] In one embodiment, in order to reduce or eliminate errors when determining the posture information of the head-mounted display device 105, the gravity direction information detected by the sensor in the head-mounted display device 105 can be further considered, and based on the gravity direction information, it can be determined whether the head-mounted display device 105 is tilted in the gravity direction, and the posture information of the head-mounted display device 105 can be corrected or rectified according to the amount of tilt.

[0059] Figure 6 A method for determining a relative posture relationship between at least two users according to an embodiment is shown, the method comprising (partially with Figure 4 The steps are similar and will not be repeated here):

[0060] Step 601: Marking the spatial position of a second user wearing a second head-mounted display device in the field of view of a first user wearing a first head-mounted display device by using the handheld smart device.

[0061] Step 602: Mark, by means of the handheld smart device, in the visual field of the first user wearing the first head-mounted display device, the spatial position at which the second user is aiming through the crosshairs on the display screen of the second head-mounted display device.

[0062] Step 603: Obtain gravity direction information detected by a sensor in the second head-mounted display device.

[0063] The head mounted display device 105 may include a device capable of detecting the direction of gravity, such as a gravity sensor, a gravity inductor, etc. Through the detected gravity direction information, it is possible to determine whether the head mounted display device 105 is currently tilted, how it is tilted, and the specific degree of tilt.

[0064] Step 604: Based on the spatial position of the second user wearing the second head-mounted display device, the spatial position of the crosshairs, and the gravity direction information, determine the relative position and posture relationship between the first user wearing the first head-mounted display device and the second user wearing the second head-mounted display device.

[0065] Once the gravity direction information of the head-mounted display device 105 is obtained, the posture information of the user 104 or its head-mounted display device 105 can be determined more accurately based on the spatial position of the user 104 or its head-mounted display device 105, the spatial position at which the crosshairs of the head-mounted display device 105 are aligned, and the gravity direction information, even when the user 104 of the head-mounted display device 105 makes some unconventional movements (for example, rotating the head significantly around the direction of sight).

[0066] In one embodiment, a common spatial coordinate system can be established, allowing one of the first head-mounted display device and the second head-mounted display device to determine its position and posture in the common spatial coordinate system, and based on the position and posture and the relative position and posture between the first head-mounted display device and the second head-mounted display device, the position and posture of the other of the first head-mounted display device and the second head-mounted display device in the common spatial coordinate system can be determined. The common spatial coordinate system refers to a spatial coordinate system that can be used by the first head-mounted display device and the second head-mounted display device. It can be, for example, a site coordinate system (e.g., a coordinate system established for a room, building, campus, etc.) or a world coordinate system. Various feasible positioning and posture determination technologies (e.g., image recognition, three-dimensional point cloud of the scene, visual markers, optical communication devices or optical tags, optical signals, wireless signals, satellite signals, etc.) can be used to determine the position and posture information of the first head-mounted display device or the second head-mounted display device in a certain spatial coordinate system.

[0067] Figure 7 A method for determining a relative posture relationship between at least two users according to an embodiment is shown, the method comprising (partially with Figure 4 The steps are similar and will not be repeated here):

[0068] Step 701: Determine the position and posture information of one of the first head-mounted display device and the second head-mounted display device in a certain spatial coordinate system.

[0069] Various feasible positioning and posture determination technologies can be used to determine the position and posture information of the head-mounted display device in a certain spatial coordinate system.

[0070] In one embodiment, the position and posture of the head-mounted display device in the spatial coordinate system can be determined based on visual markers in the space. A visual marker refers to a marker that can be recognized by an electronic device, which can have various forms. In some embodiments, the visual marker can be used to transmit information, and the information can be obtained by a smart device. For example, the visual marker can be an optical communication device that can emit coded light information, or the visual marker can be a graphic with coded information, such as a QR code (such as a QR code, a mini-program code), a barcode, etc. The head-mounted display device can obtain an image containing the visual marker by capturing an image of the visual marker through an image capture device thereon, and can identify the information conveyed by the visual marker and determine the position or posture information of the head-mounted display device relative to the visual marker by analyzing the imaging of the visual marker in the image, thereby determining the position and posture of the head-mounted display device in the spatial coordinate system.

[0071] Step 702: Mark the spatial position of the second head-mounted display device in the field of view of the first head-mounted display device by using the handheld smart device.

[0072] Step 703: Mark, by means of the handheld smart device, the spatial position where the crosshair on the display screen of the second head-mounted display device is aimed in the field of view of the first head-mounted display device.

[0073] Step 704: Determine the relative position and posture relationship between the first head-mounted display device and the second head-mounted display device based on the spatial position of the second head-mounted display device and the spatial position where the crosshair is aligned.

[0074] Step 705: Determine the position and posture information of the other one of the first head-mounted display device and the second head-mounted display device in the spatial coordinate system based on the position and posture information of one of the first head-mounted display device and the second head-mounted display device in the spatial coordinate system and the relative position and posture relationship between the first head-mounted display device and the second head-mounted display device.

[0075] Step 706: Track position changes and / or posture changes of the first head-mounted display device and the second head-mounted display device to achieve interaction between the first user and the second user.

[0076] Figure 8 The figure shows an actual effect diagram according to an embodiment, which is about the application of the solution of the present application in the doctor-patient communication scenario. In the figure, the first user (doctor) and the second user (patient) wear head-mounted display devices respectively, and the two head-mounted display devices are bound to the same coordinate system. The doctor places the patient's skull three-dimensional model at a certain position in the space through the head-mounted display device, and performs various operations on the skull three-dimensional model through a handheld smart device (such as a mobile phone), such as moving, scaling, rotating, labeling, and indicating. The patient can observe the skull three-dimensional model placed by the doctor at the corresponding position in the space through the head-mounted display device he wears, and can observe the various operations performed by the doctor on the skull three-dimensional model in real time.

[0077] In the above description, a mobile phone is used as an example, but it can be understood that the present application is not limited to mobile phones, but can be applied to other handheld smart devices, such as smart handles, controllers, etc.

[0078] The head-mounted display device in this application may be AR / VR glasses, smart glasses, AR / VR helmets, or any other glasses or helmets that can be used to present information to a user. The head-mounted display device in this application also includes glasses formed by adding components or plug-ins to ordinary optical glasses, for example, glasses formed by adding a display device to ordinary optical glasses.

[0079] In one embodiment of the present invention, the present invention may be implemented in the form of a computer program. The computer program may be stored in various storage media (e.g., a hard disk, an optical disk, a flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

[0080] In another embodiment of the present invention, the present invention may be implemented in the form of an electronic device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of the present invention can be implemented.

[0081] References herein to "various embodiments," "some embodiments," "one embodiment," or "an embodiment" refer to a particular feature, structure, or property described in connection with the embodiment as being included in at least one embodiment. Thus, the appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment" throughout this document do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in connection with one embodiment may be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable. Expressions such as "according to A," "based on A," "through A," or "using A" as used herein are intended to be non-exclusive, meaning that "according to A" may include "according only to A" or "according to A and B," unless specifically stated to mean "according only to A." For clarity, exemplary operational steps are described in a certain order throughout this application, but those skilled in the art will appreciate that not all of these operational steps are essential, and some of these steps may be omitted or replaced by other steps. These operation steps do not have to be executed sequentially in the manner shown; on the contrary, some of these operation steps can be executed in different orders or in parallel according to actual needs, as long as the new execution manner is not illogical or inoperable.

[0082] Having thus described several aspects of at least one embodiment of the present invention, it will be appreciated that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended to be within the spirit and scope of the present invention. While the present invention has been described with reference to certain embodiments, the present invention is not limited to the embodiments described herein and encompasses various changes and variations that may be made without departing from the scope of the present invention.

Claims

1. A method for determining a relative posture relationship between at least two users, wherein: A first user wears a first head-mounted display device and has a controller configured for the first head-mounted display device, a second user wears a second head-mounted display device, one or more display screens of the second head-mounted display device can display a crosshair, and the crosshair is used to aim at any object or a portion of an object in space, and the method includes: marking, by the controller, a spatial position of a second user wearing the second head-mounted display device in the visual field of the first user wearing the first head-mounted display device; marking, by the controller, in the visual field of the first user wearing the first head-mounted display device, the spatial position of the object or a part of the object aimed at by the second user through the crosshairs on the display screen of the second head-mounted display device; and Based on the spatial position of the second user wearing the second head-mounted display device and the spatial position of the object or a part of the object aligned with the crosshairs, determine the position and posture relationship of the second user wearing the second head-mounted display device relative to the first user wearing the first head-mounted display device.

2. The method according to claim 1, wherein The marking of the spatial position of the second user wearing the second head-mounted display device comprises recording position information of the spatial position of the second user wearing the second head-mounted display device relative to the first user or the first head-mounted display device worn by the first user; and Marking the spatial position of the object or a part of the object that the second user is aimed at through the crosshairs on the display screen of the second head-mounted display device includes recording the position information of the spatial position of the object or the part of the object that the crosshairs are aimed at relative to the first user or the first head-mounted display device worn by the first user.

3. The method according to claim 1, further comprising: Obtaining gravity direction information detected by a sensor in the second head-mounted display device; as well as The determining of the position and posture relationship of the second user wearing the second head-mounted display device relative to the first user wearing the first head-mounted display device based on the spatial position of the second user wearing the second head-mounted display device and the spatial position of the object or a part of the object aligned with the crosshairs includes: determining the position and posture relationship of the second user wearing the second head-mounted display device relative to the first user wearing the first head-mounted display device based on the spatial position of the second user wearing the second head-mounted display device, the spatial position of the object or a part of the object aligned with the crosshairs, and the gravity direction information.

4. The method according to claim 1, further comprising: Tracking position changes and / or posture changes of the first head-mounted display device and the second head-mounted display device to enable interaction between the first user and the second user.

5. The method according to claim 1, wherein The controller is a handheld smart device, and wherein, Controlling the movement of a virtual object in the first user's field of view by changing the spatial position of the handheld smart device to mark the spatial position of a second user wearing the second head-mounted display device in the first user's field of view; and / or The movement of the virtual object in the first user's field of view is controlled by changing the spatial position of the handheld smart device, so as to mark the spatial position of the object or a part of the object that the second user is aiming at through the crosshairs on the display screen of the second head-mounted display device in the first user's field of view.

6. The method according to claim 1, wherein First, mark the spatial position of the second user wearing the second head-mounted display device, and then mark the spatial position of the object or a part of the object that the second user is aiming at through the crosshairs on the display screen of the second head-mounted display device; or First, mark the spatial position of the object or a part of the object that the second user points at through the crosshairs on the display screen of the second head-mounted display device, and then mark the spatial position of the second user wearing the second head-mounted display device.

7. The method according to claim 1, wherein The controller includes: Handheld smart devices; User voice-based controller; A controller based on the user's gaze direction; or A controller based on user gestures.

8. The method according to claim 1, wherein The crosshair is presented on a display screen of the second head-mounted display device, and the marked spatial position of the second user wearing the second head-mounted display device is the position of the display screen presenting the crosshair, or the position of the user's eyes corresponding to the display screen.

9. The method according to claim 1, wherein The spatial position of the second user wearing the second head-mounted display device includes: a position of a left eye or a right eye of a second user wearing the second head-mounted display device; a position of a left display screen or a right display screen of the second head-mounted display device; The positions of the centers of the left display screen and the right display screen of the second head mounted display device; the position of the head of a second user wearing the second head-mounted display device; or The position of the center of the second head-mounted display device.

10. The method according to claim 1, wherein The object or a part of the object that the second user points at through the crosshairs on the display screen of the second head mounted display device is determined in the following manner: Agreed in advance by the first user and the second user; The first user informs the second user; or The second user informs the first user.

11. A storage medium storing a computer program, wherein when the computer program is executed by a processor, the computer program can be used to implement the method according to any one of claims 1 to 10.

12. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the computer program can be used to implement the method according to any one of claims 1 to 10.

13. An interactive system comprising a controller and a head-mounted display device, wherein the controller and the head-mounted display device can be used to implement the method according to any one of claims 1 to 10.

Images