Interaction method and system based on handheld smart device and head-mounted display device

Abstract

The application relates to an interactive method and system based on a handheld smart device and a head-mounted display device, wherein the handheld smart device is provided with an image acquisition device, the method comprises the following steps: collecting a head image of a user wearing the head-mounted display device through the image acquisition device of the handheld smart device; analyzing the head image; determining the relative position between the handheld smart device and the head-mounted display device according to the analysis result of the head image; tracking the position change of the handheld smart device and the posture change of the head-mounted display device, and determining the real-time relative position between the handheld smart device and the head-mounted display device; and presenting virtual content associated with the handheld smart device on a corresponding area on the display medium of the head-mounted display device according to the real-time relative position between the handheld smart device and the head-mounted display device.

Description

Technical Field

[0001] This invention relates to the field of information interaction, and in particular to an interaction method and system based on a handheld smart device and a head-mounted display device. Background Technology

[0002] The statements in this section are merely to provide background information related to the technical solutions of this application to aid understanding, and do not necessarily constitute prior art for the technical solutions of this application.

[0003] In recent years, augmented reality (AR) technology has gained increasingly widespread application. Currently, the most commonly used display and interaction devices in AR applications are handheld smart devices (such as mobile phones) and head-mounted display devices (such as AR glasses and smart glasses). Virtual content can be presented to the user on the screen of a mobile phone or smart glasses, and the user can interact with this virtual content. Virtual content can be, for example, icons, images, text, emojis, virtual people, virtual 3D objects, 3D scene models, an animation, a video, and so on. In AR applications, the mobile phone and smart glasses determine their position and / or pose information (hereinafter collectively referred to as "pose information") in real time, and determine which virtual content to present and how to present it based on this position and / or pose information.

[0004] However, since consumer-grade AR glasses are not yet widespread, users typically use either a mobile phone or a head-mounted display device for information interaction in AR applications. For example, in open business environments, users usually use mobile phones for information interaction, while in relatively closed industrial operation and maintenance environments, users usually use head-mounted displays for information interaction.

[0005] As consumer-grade AR glasses become increasingly popular, smartphones and AR glasses will become the two most important smart devices that users carry with them at all times. Therefore, it is necessary to consider how to use smartphones and head-mounted display devices (such as AR glasses) simultaneously to achieve information interaction and leverage the advantages of both. Summary of the Invention

[0006] One aspect of the present invention relates to an interaction method based on a handheld smart device and a head-mounted display device, wherein the handheld smart device includes an image acquisition device, and the method includes: acquiring a head image of a user wearing the head-mounted display device using the image acquisition device of the handheld smart device; analyzing the head image; determining the relative position between the handheld smart device and the head-mounted display device based on the analysis result of the head image; tracking position changes of the handheld smart device and posture changes of the head-mounted display device to determine the real-time relative position between the handheld smart device and the head-mounted display device; and displaying virtual content associated with the handheld smart device in a corresponding area on the display medium of the head-mounted display device based on the real-time relative position between the handheld smart device and the head-mounted display device.

[0007] In one embodiment, the method further includes: tracking positional changes of the head-mounted display device.

[0008] In one embodiment, the method further includes: determining the relative posture between the handheld smart device and the head-mounted display device based on the analysis results of the head image; tracking the posture changes of the handheld smart device; determining the real-time relative posture between the handheld smart device and the head-mounted display device; and wherein, presenting virtual content associated with the handheld smart device in a corresponding area on the display medium of the head-mounted display device according to the real-time relative position between the handheld smart device and the head-mounted display device includes: presenting virtual content associated with the handheld smart device in a corresponding area on the display medium of the head-mounted display device according to the real-time relative position and posture between the handheld smart device and the head-mounted display device.

[0009] In one embodiment, the position or posture changes of the handheld smart device and the head-mounted display device are tracked by sensors of their respective devices.

[0010] In one embodiment, determining the relative position between the handheld smart device and the head-mounted display device includes: representing the position of the head-mounted display device in a spatial coordinate system defined based on the handheld smart device; or, representing the position of the handheld smart device in a spatial coordinate system defined based on the head-mounted display device.

[0011] In one embodiment, determining the relative posture between the handheld smart device and the head-mounted display device includes: representing the posture of the head-mounted display device in a spatial coordinate system defined based on the handheld smart device; or, representing the posture of the handheld smart device in a spatial coordinate system defined based on the head-mounted display device.

[0012] In one embodiment, the method further includes: determining the position and orientation of one of the handheld smart device and the head-mounted display device in a spatial coordinate system; and determining the position and orientation of the other of the handheld smart device and the head-mounted display device in the spatial coordinate system based on the position and orientation and the relative position and relative orientation between the handheld smart device and the head-mounted display device determined based on the head image.

[0013] In one embodiment, the method further includes: determining the position and orientation of the handheld smart device and the head-mounted display device in a certain spatial coordinate system; and correcting the position and orientation of at least one of the handheld smart device and the head-mounted display device in the spatial coordinate system based on the relative position and relative orientation between the handheld smart device and the head-mounted display device determined based on the head image.

[0014] In one embodiment, the method further includes: determining the position and orientation of the handheld smart device and the head-mounted display device in a certain spatial coordinate system; and correcting the position and orientation of the handheld smart device in the spatial coordinate system based on the relative position and relative orientation between the handheld smart device and the head-mounted display device determined based on the head image and the position and orientation of the head-mounted display device in the spatial coordinate system.

[0015] Another aspect of the invention relates to a storage medium storing a computer program that, when executed by a processor, can be used to implement the methods described in this application.

[0016] Another aspect of the present invention relates to an electronic device including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, can be used to implement the methods described in this application.

[0017] Another aspect of the present invention relates to an interactive system comprising a handheld smart device and a head-mounted display device, the handheld smart device having an image acquisition device, and the handheld smart device and the head-mounted display device being capable of implementing the methods described in this application.

[0018] The solution proposed in this application unifies handheld smart devices and head-mounted display devices into a single coordinate system. This allows the handheld smart devices to be primarily used for interactive operations, while the head-mounted display devices are primarily used for information presentation. This approach fully leverages the advantages of both devices, and their collaborative operation enables convenient information display and interactive functions.

[0019] On the other hand, using the image acquisition device of the handheld smart device to acquire the user's head image to determine the relative pose between the handheld smart device and the head-mounted display device also has the following advantages: (1) It reduces the workload of the head-mounted display device, reduces its power consumption, and improves its battery life; (2) The user's head is larger than the handheld smart device and has obvious three-dimensional features, thus providing more accurate relative positioning and pose determination capabilities; (3) During the user experience, the user's head or the head-mounted smart device worn on it is generally not obstructed, while the handheld smart device is often obstructed by the user's hand, which makes the recognition and positioning based on the user's head image more stable; (4) The user's head shape is relatively fixed, while the user's hand shape often changes during operation, which makes the recognition and positioning based on the user's head image more reliable; (5) The wearing position of the head-mounted smart device on the user's head will not change much (similar to a rigid connection), so the relative position or pose between the handheld smart device and the head-mounted display device can be accurately determined based on the user's head image. Attached Figure Description

[0020] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0021] Figure 1 An interactive system based on a mobile phone and a head-mounted display device according to one embodiment is shown;

[0022] Figure 2 An interaction method based on a mobile phone and a head-mounted display device according to one embodiment is illustrated;

[0023] Figure 3 An interaction method based on a mobile phone and a head-mounted display device according to another embodiment is shown;

[0024] Figure 4 An interactive interface according to one embodiment is shown;

[0025] Figure 5 A schematic diagram is shown representing the position and orientation of a mobile phone in a spatial coordinate system defined based on a head-mounted display device, according to one embodiment;

[0026] Figure 6 An interaction method based on a mobile phone and a head-mounted display device according to another embodiment is illustrated; and

[0027] Figure 7 An interaction method based on a mobile phone and a head-mounted display device according to another embodiment is shown.

[0028] Figure 8 An actual effect diagram according to one embodiment is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] Figure 1 An interactive system based on a mobile phone and a head-mounted display device (HUD) according to one embodiment is illustrated. The system includes a mobile phone 102 belonging to a user 101 and a head-mounted display device (e.g., AR glasses) 103. The user 101 is positioned in a scene, holding the mobile phone 102 in their hand and wearing the head-mounted display device 103, and can experience various augmented reality (AR) or virtual reality (VR) applications through the phone 102 and the HUD 103. In this system, the mobile phone 102 is primarily used to implement interactive functions in various AR or VR applications, such as selecting virtual objects, manipulating virtual objects, inputting information, etc. The head-mounted display device 103 has a display medium, which is primarily used to present image or video information from AR or VR applications to the user 101, for example, presenting various virtual objects from AR or VR applications to the user 101.

[0031] In one embodiment, images or video information from augmented reality or virtual reality applications can also be presented on the mobile phone 102 (e.g., on the display screen of the mobile phone 102). In another embodiment, the user 101 can also use the head-mounted display device 103 to perform information interaction functions in augmented reality or virtual reality applications, such as through voice, buttons, touch bars, gestures, eye contact, etc.

[0032] The mobile phone 102 has one or more image acquisition devices (e.g., cameras) that can be used to capture images of the head of the user 101 wearing the head-mounted display device 103. The image acquisition devices may be located on the front, back, or other locations of the mobile phone 102. In one embodiment, the head-mounted display device 103 may also have one or more image acquisition devices.

[0033] To experience various augmented reality or virtual reality applications through smart devices (which can be mobile phones or head-mounted smart devices), it is typically necessary to determine the smart device's position and / or attitude information in space. As those skilled in the art will understand, various feasible positioning / attitude determination techniques can be used to determine the smart device's position and / or attitude information in space, such as image recognition, scene 3D point clouds, visual markers, optical communication devices or optical tags, optical signals, wireless signals, satellite signals, etc. Simultaneously, the smart device can also use various built-in sensors (e.g., accelerometers, magnetometers, orientation sensors, gravity sensors, gyroscopes, cameras, etc.) and methods known in the art (e.g., inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.) to measure or track its position changes and / or attitude changes, thereby determining the smart device's real-time position and / or attitude in space.

[0034] Figure 2 An interaction method based on a mobile phone and a head-mounted display device according to one embodiment is illustrated. In this method, by unifying the mobile phone and the head-mounted display device into a single coordinate system, the mobile phone is mainly used for interactive operations, and the head-mounted display device is mainly used for information presentation, thereby fully utilizing the advantages of each device and enabling them to work together to achieve convenient information display and interaction functions. The method includes:

[0035] Step 201: Acquire a head image of the user wearing the head-mounted display device using the image acquisition device of the mobile phone.

[0036] In this method, users can wear head-mounted display devices to experience augmented reality or virtual reality applications. At the same time, users can use their mobile phones to perform various operations required in the augmented reality or virtual reality applications, such as clicking on operation menus, clicking on operation buttons, aiming at virtual objects, clicking on virtual objects, moving virtual objects, drawing virtual works, inputting information, and so on.

[0037] During the operation of the mobile phone, the user can change the position and / or posture of the phone. In this process, the image acquisition device on the phone can capture an image of the user's head while wearing the head-mounted display device. The process of capturing the user's head image can be imperceptible to the user.

[0038] It's understandable that a mobile phone doesn't necessarily need to continuously capture images of the user's head; it can capture images only when an image of the user's head is detected. For example, the phone's image capture device can periodically or frequently detect whether a user's head is in its field of view, and if detected, then capture an image of the user's head.

[0039] The user's head image may include the user's head or a part of the head (e.g., the user's face, nose, eyes, ears, etc.), or an object worn on the user's head (e.g., a head-mounted display device) or a part thereof, as long as the position and / or posture information of the user's head or the head-mounted display device relative to the mobile phone can be analyzed from the head image.

[0040] Step 202: Analyze the head image.

[0041] After acquiring the user's head image, the image is analyzed. For example, the size, posture, and imaging position of the user's head or a portion thereof can be analyzed, as well as the size, posture, and imaging position of any object worn on the user's head (e.g., a head-mounted display device). This analysis can be performed by a mobile phone, but it can also be performed by the head-mounted display device or other devices (by receiving the head image from the mobile phone). How to perform this analysis on the user's head image is well known to those skilled in the art and will not be elaborated upon here.

[0042] Step 203: Based on the analysis results of the head image, determine the relative position between the mobile phone and the head-mounted display device.

[0043] Based on the analysis of the user's head image, the relative position between the mobile phone and the head-mounted display device can be determined. For example, by analyzing the size of the user's head, the distance between the phone and the user's head can be determined, and since the head-mounted display device is usually worn in a fixed position on the user's head, the relative distance between the phone and the head-mounted display device can be determined. Similarly, by analyzing the user's head posture, the relative direction between the phone and the head-mounted display device can be determined. For example, if the captured head image is of the user's frontal view, it indicates that the phone is located directly in front of the user's head. Thus, the relative positional relationship between the phone and the head-mounted display device can be determined. The relative position between the phone and the head-mounted display device could be, for example, the phone being directly in front of the head-mounted display device at a distance of 30 cm, the phone being 15 degrees to the right of the head-mounted display device at a distance of 20 cm, the phone being 10 degrees downwards of the head-mounted display device at a distance of 25 cm, and so on. These relative positions can be represented in various existing spatial coordinate systems.

[0044] In one embodiment, the relative position between the mobile phone and the user's head can be determined first based on the analysis results of the head image. Then, the relative position between the mobile phone and the head-mounted display device can be determined based on the wearing position of the head-mounted display device on the user's head. In another embodiment, the relative position between the mobile phone and the head-mounted display device can be determined directly based on the analysis results of the head image. For example, the relative position between the mobile phone and the head-mounted display device can be determined based on the size, posture, and imaging position of the head-mounted display device in the head image.

[0045] Step 204: Track the position changes of the mobile phone and the posture changes of the head-mounted display device to determine the real-time relative position between the mobile phone and the head-mounted display device.

[0046] In one embodiment, after determining the relative position between the mobile phone and the head-mounted display device, the positional changes of the mobile phone and the posture changes of the head-mounted display device can be tracked, and the real-time relative position between the mobile phone and the head-mounted display device can be determined, such as the current direction and distance of the mobile phone relative to the head-mounted display device.

[0047] In one embodiment, the position and / or attitude changes of the mobile phone and the head-mounted display device can be measured or tracked by their respective sensors (e.g., accelerometer, magnetometer, orientation sensor, gravity sensor, gyroscope, camera, etc.) using methods known in the art (e.g., inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.) to determine the real-time position and / or attitude of the mobile phone and the head-mounted display device in space.

[0048] In this embodiment, only the posture changes of the head-mounted display device are tracked because many low- to mid-range head-mounted display devices currently only have 3DOF posture tracking capabilities. Therefore, in this embodiment, the positional changes of the head-mounted display device can be disregarded or ignored. Although the positional changes of the head-mounted display device are not considered in this embodiment, for some augmented reality or virtual reality applications where users typically experience them in one place or a specific small area (i.e., users usually only need to turn their heads or bodies to experience them), this will not have a significant adverse impact on the user experience.

[0049] In another embodiment, the positional changes of the head-mounted display can be further tracked; that is, a head-mounted display with 6DOF position and pose tracking capabilities can be used. Because this embodiment can track the positional changes of the head-mounted display (which is also, in effect, the user's positional changes), it is also applicable to augmented reality or virtual reality applications where users typically move over a large area, such as multiplayer shooting games within a single arena.

[0050] Step 205: Based on the real-time relative position between the mobile phone and the head-mounted display device, virtual content associated with the mobile phone is displayed in the corresponding area on the display medium of the head-mounted display device.

[0051] Once the real-time relative position between the mobile phone and the head-mounted display device is determined, virtual content associated with the mobile phone can be displayed on the corresponding area of ​​the head-mounted display device's display medium based on this relative position. For example, in augmented reality or virtual reality applications, when using a mobile phone to virtually paint in space, a virtual paintbrush or content drawn by the mobile phone can be displayed on the corresponding area of ​​the head-mounted display device's display medium based on the real-time relative position between the mobile phone and the head-mounted display device. The virtual content associated with the mobile phone can be displayed overlapping with the mobile phone, around the mobile phone (e.g., above the mobile phone, in front of the mobile phone, etc.), or in other locations as needed.

[0052] In one embodiment, the mobile phone can be further used to perform certain operations, such as clicking the phone screen or pressing a button to perform certain operations, such as firing bullets, placing virtual objects, deleting virtual objects, etc.

[0053] In one embodiment, in addition to determining the relative position between the mobile phone and the head-mounted display device, their relative orientation can also be determined. Thus, during information interaction, not only the phone's position can be utilized, but its orientation can also be further utilized; for example, rotating the phone can change its orientation to correspondingly alter the firing direction of weapons in a game. Figure 3 An interaction method based on a mobile phone and a head-mounted display device according to another embodiment is shown, the method including (partial steps and) Figure 2 The steps are similar and will not be repeated here:

[0054] Step 301: Acquire a head image of the user wearing the head-mounted display device using the image acquisition device of the mobile phone.

[0055] Step 302: Analyze the head image.

[0056] Step 303: Based on the analysis results of the head image, determine the relative position and orientation between the mobile phone and the head-mounted display device.

[0057] In this embodiment, in addition to determining the relative position between the mobile phone and the head-mounted display device based on the analysis results of the user's head image, the relative posture between the mobile phone and the head-mounted display device can also be determined based on the analysis results of the head image. For example, the posture information of the mobile phone relative to the head-mounted display device can be determined based on the imaging position of the user's head in the image.

[0058] In one embodiment, the relative position and posture of the mobile phone and the user's head can be determined first based on the analysis results of the head image. Then, based on the wearing position of the head-mounted display device on the user's head, the relative position and posture between the mobile phone and the head-mounted display device can be determined. In another embodiment, the relative position and posture between the mobile phone and the head-mounted display device can be determined directly based on the analysis results of the head image. For example, the relative position and posture between the mobile phone and the head-mounted display device can be determined based on the size, posture, and imaging position of the head-mounted display device in the head image.

[0059] Step 304: Track the position and attitude changes of the mobile phone and the head-mounted display device to determine the real-time relative position and attitude between the mobile phone and the head-mounted display device.

[0060] After determining the relative position and orientation between the mobile phone and the head-mounted display device, the changes in their position and orientation can be tracked to determine their real-time relative position and orientation. In one embodiment, the changes in position and orientation of the mobile phone and the head-mounted display device can be tracked using their respective sensors.

[0061] Step 305: Based on the real-time relative position and posture between the mobile phone and the head-mounted display device, present virtual content associated with the mobile phone in the corresponding area on the display medium of the head-mounted display device.

[0062] Once the real-time relative position and orientation between the mobile phone and the head-mounted display device are determined, virtual content associated with the mobile phone can be displayed on the corresponding area of ​​the display medium of the head-mounted display device based on the relative position and orientation.

[0063] In the above embodiments, the position and attitude changes of both the mobile phone and the head-mounted display device were tracked. However, it is understood that in one embodiment, the position and attitude changes of the mobile phone may also be tracked, but only the attitude changes of the head-mounted display device are tracked (i.e., for head-mounted display devices that only have 3DOF attitude tracking capabilities).

[0064] Figure 4An interactive interface according to one embodiment is shown, which is observed by a user through the display medium of a head-mounted display device. In this interactive interface, there is a virtual object (e.g., a bow and arrow for a shooting game) 401, the user's hand 402, and a mobile phone 403. For augmented reality (AR) applications, the hand 402 and mobile phone 403 can be actual objects observed by the user through the display medium of the head-mounted display device (e.g., lenses), while the virtual object 401 is a graphic or image rendered on the display medium of the head-mounted display device. For virtual reality (VR) applications, the virtual object 401, hand 402, and mobile phone 403 can all be graphics or images rendered on the display medium of the head-mounted display device, and their rendering positions can depend on the actual positions of the user's hand and the actual mobile phone. In one embodiment, for virtual reality (VR) applications, only the virtual object 401 may be presented without the hand 402 and mobile phone 403, or the hand 402 and virtual object 401 may be presented without the mobile phone 403. It is understood that the interactive interface may also have other elements.

[0065] In this interactive interface, the real-time position and / or posture of the virtual object (e.g., a bow and arrow for a shooting game) 401 is determined based on the real-time position and / or posture of the actual mobile phone, and the virtual object 401 is rendered on the display medium of the head-mounted display device according to its real-time position and / or posture.

[0066] In one embodiment, a first virtual object (e.g., a virtual interactive menu) can be displayed in space at a certain position and orientation, while the user can interact with the first virtual object by moving or rotating the phone to change the position and orientation of a second virtual object (e.g., a virtual ray used for selection in the interactive menu).

[0067] In one embodiment, when determining the relative position and / or orientation between the mobile phone and the head-mounted display device, the position and / or orientation of the mobile phone can be represented in a spatial coordinate system defined by the head-mounted display device, or the position and / or orientation of the head-mounted display device can be represented in a spatial coordinate system defined by the mobile phone.

[0068] Figure 5A schematic diagram illustrating the representation of a mobile phone's position and orientation in a spatial coordinate system defined based on a head-mounted display device, according to one embodiment, is shown. In this diagram, the spatial coordinate system with origin O is defined based on the head-mounted display device, while another spatial coordinate system with origin S is shown, used to represent the mobile phone's position and orientation within the head-mounted display device-defined spatial coordinate system. Specifically, a spatial coordinate system (origin O) can be established based on the head-mounted display device, while another coordinate system (origin S) can be established based on the mobile phone. Based on the determination of the relative position and orientation relationship between the head-mounted display device and the mobile phone, the mobile phone's position and orientation can be conveniently represented in the spatial coordinate system established based on the head-mounted display device, or vice versa.

[0069] In one embodiment, a common spatial coordinate system can be established, allowing one of the mobile phone and the head-mounted display device to determine its position and orientation within this common spatial coordinate system. Based on this position and orientation, and the relative position and orientation between the mobile phone and the head-mounted display device, the position and orientation of the other mobile phone and the head-mounted display device within the same common spatial coordinate system can be determined. The common spatial coordinate system refers to a spatial coordinate system that can be shared by both the mobile phone and the head-mounted display device. This can be, for example, a location coordinate system (e.g., a coordinate system established for a room, building, or park) or a world coordinate system. Various feasible positioning and orientation technologies (e.g., image recognition, scene 3D point clouds, visual markers, optical communication devices or optical tags, optical signals, wireless signals, satellite signals, etc.) can be used to determine the position and orientation information of the mobile phone or head-mounted display device within a given spatial coordinate system.

[0070] Figure 6 An interaction method based on a mobile phone and a head-mounted display device, according to another embodiment, is illustrated, in which the mobile phone determines its position and orientation in a common spatial coordinate system. The method includes (partial steps of...) Figure 2 or Figure 3 The steps are similar and will not be repeated here:

[0071] Step 601: The mobile phone determines its position and orientation in the public spatial coordinate system.

[0072] Mobile phones can use various feasible positioning and orientation technologies to determine their position and orientation in a certain spatial coordinate system.

[0073] In one embodiment, a mobile phone can determine its position and orientation in a spatial coordinate system based on visual signs in space. A visual sign is a sign that can be recognized by an electronic device and can take various forms. In some embodiments, a visual sign can be used to convey information that can be obtained by a smart device (e.g., a mobile phone, smart glasses, etc.). For example, a visual sign can be an optical communication device capable of emitting coded light information, or it can be a graphic with coded information, such as a QR code (e.g., a QR code, a mini-program code), a barcode, etc. The mobile phone can acquire an image containing the visual sign by using its image acquisition device, and can identify the information conveyed by the visual sign and determine the position or orientation information of the mobile phone relative to the visual sign by analyzing the imaging of the visual sign in the image, thereby determining the position and orientation of the mobile phone in the spatial coordinate system.

[0074] Step 602: Acquire a head image of the user wearing the head-mounted display device using the image acquisition device of the mobile phone.

[0075] Step 603: Analyze the head image.

[0076] Step 604: Based on the analysis results of the head image, determine the relative position and orientation between the mobile phone and the head-mounted display device.

[0077] Step 605: Determine the position and orientation of the head-mounted display device in the common spatial coordinate system based on the position and orientation of the mobile phone and the relative position and orientation between the mobile phone and the head-mounted display device.

[0078] In this way, even if the head-mounted display device does not have the ability to independently determine its position and orientation in the spatial coordinate system, for example, it does not have software or hardware for recognizing visual signs, or software or hardware for wireless positioning, it can still indirectly determine its position and orientation in the spatial coordinate system.

[0079] Step 606: Track the position and attitude changes of the mobile phone and the head-mounted display device to determine the real-time relative position and attitude between the mobile phone and the head-mounted display device.

[0080] Step 607: Based on the real-time relative position and posture between the mobile phone and the head-mounted display device, present virtual content associated with the mobile phone in the corresponding area on the display medium of the head-mounted display device.

[0081] In one embodiment, the head-mounted display device can also determine its position and orientation in a common spatial coordinate system, and based on that position and orientation, as well as the relative position and orientation between the mobile phone and the head-mounted display device, determine the position and orientation of the mobile phone in the common spatial coordinate system. This method is similar to... Figure 6 The methods shown are similar and will not be repeated here.

[0082] In one embodiment, the mobile phone and the head-mounted display device can each determine their respective positions and orientations in a common spatial coordinate system, and correct the position and orientation of at least one of the mobile phone and the head-mounted display device in the common spatial coordinate system based on the relative position and relative orientation between the mobile phone and the head-mounted display device.

[0083] Figure 7 An interaction method based on a mobile phone and a head-mounted display device, according to another embodiment, is illustrated, in which the mobile phone and the head-mounted display device respectively determine their positions and orientations in a common spatial coordinate system. The method includes (partial steps of...) Figure 2 or Figure 3 The steps are similar and will not be repeated here:

[0084] Step 701: The mobile phone and the head-mounted display device respectively determine their position and orientation in the common spatial coordinate system.

[0085] Step 702: Acquire a head image of the user wearing the head-mounted display device using the image acquisition device of the mobile phone.

[0086] Step 703: Analyze the head image.

[0087] Step 704: Based on the analysis results of the head image, determine the relative position and orientation between the mobile phone and the head-mounted display device.

[0088] Step 705: Correct the position and orientation of at least one of the mobile phone and the head-mounted display device in the common spatial coordinate system based on the relative position and orientation between the mobile phone and the head-mounted display device.

[0089] When determining their position and orientation in a common spatial coordinate system, mobile phones or head-mounted displays may have significant errors. However, determining their relative position and orientation based on the head image typically offers higher accuracy and reliability (due to the closer distance). Therefore, the position and orientation of at least one of the mobile phone and head-mounted display devices in the common spatial coordinate system can be corrected based on the relative position and orientation. For example, if the mobile phone has higher positioning and orientation accuracy (e.g., due to better-performing sensors and processors), the position and orientation of the head-mounted display device can be corrected based on the mobile phone's position and orientation, as well as the relative position and orientation. Similarly, if the head-mounted display device has higher positioning and orientation accuracy (e.g., due to better-performing sensors and processors), the position and orientation of the mobile phone can be corrected based on the head-mounted display device's position and orientation, as well as the relative position and orientation. It is understood that the position and orientation of both the mobile phone and the head-mounted display device can also be corrected simultaneously.

[0090] Step 706: Track the position and attitude changes of the mobile phone and the head-mounted display device to determine the real-time relative position and attitude between the mobile phone and the head-mounted display device.

[0091] Step 707: Based on the real-time relative position and posture between the mobile phone and the head-mounted display device, present virtual content associated with the mobile phone in the corresponding area on the display medium of the head-mounted display device.

[0092] By using a common spatial coordinate system, the solution in this application can track the position and orientation information of each participating user's mobile phone and head-mounted display device in that spatial coordinate system, thereby enabling two or more users to participate in the same scene simultaneously, such as two or more users experiencing a multiplayer shooting game at the same time.

[0093] In one embodiment, as needed, the image acquisition device of the mobile phone can re-acquire an image of the user's head wearing the head-mounted display device, analyze the head image, and, based on the analysis results, redetermine the relative position and / or orientation between the mobile phone and the head-mounted display device. For example, this operation can be performed when the user's head appears within the field of view of the mobile phone's image acquisition device. After determining the relative position and / or orientation between the mobile phone and the head-mounted display device, the position and / or orientation of at least one of the mobile phone and the head-mounted display device can be corrected.

[0094] In most cases, users' heads do not typically perform drastic movements or rotations, while their hands frequently engage in various complex or rapid actions. This makes the pose determination or tracking performance of head-mounted smart devices generally better than that of mobile phones. Therefore, in one embodiment, after determining the relative position and / or posture between the mobile phone and the head-mounted display device, the position and / or posture of the mobile phone in the spatial coordinate system can be corrected based on the relative position and / or posture and the position and / or posture of the head-mounted display device in the spatial coordinate system, thereby significantly reducing or eliminating the position and / or posture error of the mobile phone. As those skilled in the art will understand, various feasible positioning / posture determination techniques can be used to determine the position and / or posture of the mobile phone or head-mounted display device in the spatial coordinate system. For example, this can be achieved through image recognition, scene 3D point clouds, visual markers, optical communication devices or optical tags, optical signals, wireless signals, satellite signals, etc., and / or through various sensors built into the mobile phone or head-mounted display device to measure or track its position changes and / or posture changes, thereby determining its real-time position and / or posture in the spatial coordinate system.

[0095] In one embodiment, the relative position and / or orientation between the mobile phone and the head-mounted display device includes the position and / or orientation of the mobile phone relative to the head-mounted display device. In another embodiment, the relative position and / or orientation between the mobile phone and the head-mounted display device includes the position and / or orientation of the head-mounted display device relative to the mobile phone.

[0096] Figure 8 The illustration shows a practical effect of an embodiment of an augmented reality shooting game, in which a user holds a mobile phone and controls the position and aiming direction of a virtual pistol by changing the phone's position and orientation. After aiming, the user can fire bullets by performing actions on the phone (such as tapping the phone screen or pressing a button).

[0097] Using the image acquisition device of a mobile phone to acquire the user's head image to determine the relative pose between the mobile phone and the head-mounted display device has the following advantages: (1) It reduces the workload of the head-mounted display device, reduces its power consumption, and improves its battery life; (2) The user's head is larger than the mobile phone and has obvious three-dimensional features, thus providing more accurate relative positioning and pose determination capabilities; (3) During the user experience, the user's head or the head-mounted smart device worn on it is generally not obstructed, while the mobile phone is often obstructed by the user's hand, which makes the recognition and positioning based on the user's head image more stable; (4) The user's head shape is relatively fixed, while the user's hand shape often changes during operation, which makes the recognition and positioning based on the user's head image more reliable; (5) The wearing position of the head-mounted smart device on the user's head does not change much (similar to a rigid connection), so the relative position or pose between the handheld smart device and the head-mounted display device can be accurately determined based on the head image.

[0098] The above explanation uses a mobile phone as an example, but it is understood that this application is not limited to mobile phones, but can be applied to other handheld smart devices with image acquisition devices.

[0099] The head-mounted display device in this application can be AR glasses, VR glasses, smart glasses, or any other glasses capable of presenting information to a user. The head-mounted display device in this application also includes glasses formed by adding components or inserts to ordinary optical glasses, for example, glasses formed by adding a display device to ordinary optical glasses.

[0100] In one embodiment of the present invention, the invention can be implemented in the form of a computer program. The computer program can be stored in various storage media (e.g., hard disk, optical disk, flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

[0101] In another embodiment of the invention, the invention can be implemented as an electronic device. This electronic device includes a processor and a memory, in which a computer program is stored. When executed by the processor, the computer program can be used to implement the method of the invention.

[0102] References to “various embodiments,” “some embodiments,” “one embodiment,” or “embodiment” throughout this document refer to a particular feature, structure, or property described in connection with said embodiment that is included in at least one embodiment. Therefore, the appearance of phrases such as “in various embodiments,” “in some embodiments,” “in one embodiment,” or “in an embodiment” throughout this document does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or property can be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that such combination is not illogical or inoperable. Expressions such as “according to A,” “based on A,” “by A,” or “using A” appearing throughout this document are non-exclusive; that is, “according to A” can cover “according to A only” or “according to A and B,” unless specifically stated otherwise. In this application, some illustrative operational steps are described in a certain order for clarity, but those skilled in the art will understand that each of these operational steps is not essential, and some steps can be omitted or replaced by others. These steps do not necessarily have to be performed sequentially as shown. Instead, some of these steps can be performed in different orders or in parallel as needed, as long as the new execution method is not illogical or ineffective.

[0103] This has described several aspects of at least one embodiment of the invention. It will be understood that various changes, modifications, and improvements will be readily apparent to those skilled in the art. Such changes, modifications, and improvements are intended within the spirit and scope of the invention. While the invention has been described through some embodiments, it is not limited to the embodiments described herein, and includes various changes and variations made without departing from the scope of the invention.

Claims

1. An interaction method based on a handheld smart device and a head-mounted display device, wherein, The handheld smart device has an image acquisition device, and the method includes: The handheld smart device acquires an image of the user's head while wearing the head-mounted display device; Analyze the head image; Based on the analysis results of the head image, the relative position between the handheld smart device and the head-mounted display device is determined; Track the positional changes of the handheld smart device and the posture changes of the head-mounted display device to determine the real-time relative position between the handheld smart device and the head-mounted display device; and Based on the real-time relative position between the handheld smart device and the head-mounted display device, virtual content associated with the handheld smart device is displayed in a corresponding area on the display medium of the head-mounted display device.

2. The method according to claim 1, further comprising: Track the positional changes of the head-mounted display device.

3. The method according to claim 1, further comprising: Based on the analysis results of the head image, the relative posture between the handheld smart device and the head-mounted display device is determined; Track the posture changes of the handheld smart device; Determine the real-time relative posture between the handheld smart device and the head-mounted display device; And among them, The step of presenting virtual content associated with the handheld smart device in a corresponding area on the display medium of the head-mounted display device based on the real-time relative position between the handheld smart device and the head-mounted display device includes: Based on the real-time relative position and posture between the handheld smart device and the head-mounted display device, virtual content associated with the handheld smart device is presented in a corresponding area on the display medium of the head-mounted display device.

4. The method according to any one of claims 1-3, wherein, The position or posture changes of the handheld smart device and the head-mounted display device are tracked by their respective sensors.

5. The method according to claim 1, wherein, Determining the relative position between the handheld smart device and the head-mounted display device includes: The position of the head-mounted display device is represented in a spatial coordinate system defined based on the handheld smart device; or The position of the handheld smart device is represented in the spatial coordinate system defined based on the head-mounted display device.

6. The method according to claim 3, wherein, Determining the relative posture between the handheld smart device and the head-mounted display device includes: The posture of the head-mounted display device is represented in a spatial coordinate system defined based on the handheld smart device; or The posture of the handheld smart device is represented in a spatial coordinate system defined based on the head-mounted display device.

7. The method according to claim 3, further comprising: One of the handheld smart device and the head-mounted display device determines its position and orientation in a common spatial coordinate system; as well as Based on the position and posture, as well as the relative position and posture between the handheld smart device and the head-mounted display device determined based on the head image, the position and posture of the other of the handheld smart device and the head-mounted display device in the spatial coordinate system are determined.

8. The method according to claim 3, further comprising: The handheld smart device and the head-mounted display device respectively determine their positions and orientations in a common spatial coordinate system; as well as Based on the relative position and relative posture between the handheld smart device and the head-mounted display device determined based on the head image, the position and posture of at least one of the handheld smart device and the head-mounted display device in the spatial coordinate system are corrected.

9. The method according to claim 3, further comprising: The handheld smart device and the head-mounted display device respectively determine their positions and orientations in a common spatial coordinate system; as well as Based on the relative position and relative posture between the handheld smart device and the head-mounted display device determined from the head image, and the position and posture of the head-mounted display device in the spatial coordinate system, the position and posture of the handheld smart device in the spatial coordinate system are corrected.

10. A storage medium storing a computer program that, when executed by a processor, can be used to implement the method of any one of claims 1-9.

11. An electronic device comprising a processor and a memory, the memory storing a computer program that, when executed by the processor, is capable of implementing the method of any one of claims 1-9.

12. An interactive system comprising a handheld smart device and a head-mounted display device, the handheld smart device having an image acquisition device, and the handheld smart device and the head-mounted display device being capable of implementing the method of any one of claims 1-9.

Images