Method and system for automatically guiding the head orientation of a local user in augmented reality to realign their field of vision with remote users

Abstract

A method for guiding the head orientation of a local user wearing an immersive HMD headset in the context of remote collaboration in augmented reality, said immersive headset comprising a camera capturing a scene and transmitting the captured image to at least one remote user, said method being characterized in that it includes a step of automatically displaying an interactive graphic element prompting the local user to reorient their head, and therefore their headset, when a discrepancy is detected between what said local user sees and what remote users see; said discrepancy being able to tolerate a margin of error defined with respect to an area of ​​interest in the observed scene; the graphic element evolving according to the head movements of the local user and disappearing when the discrepancy disappears, in other words, when the scene observed by the remote users corresponds to the scene viewed by the local user.

Description

Title of the invention: Method and system for automatically guiding the orientation of the head of a local user in augmented reality to realign the field of vision with remote users Technical field

[0001] The present invention belongs to the general field of augmented reality (AR), in particular remote collaboration in augmented reality, and relates more particularly to a method and a system for automatically guiding the orientation of the head of a local user in augmented reality to realign what said local user sees with what remote users see.

[0002] The present invention finds a direct, but not exclusive, application in remote assistance and / or training in industry or in the medical field (augmented reality surgery). State of the art

[0003] Augmented reality (AR) offers considerable advantages for remote collaboration scenarios. However, when using a head-mounted display (HMD), remote users do not always see exactly what local users are looking at. This occurs when there is a spatial mismatch between the center of the HMD's cameras' field of view and the center of the user's field of view. Such a mismatch can limit the ability of remote users to see objects of interest to them, creating confusion and hindering collaboration.

[0004] Indeed, seeing what other users are seeing, doing, and in what context is a key feature of collaboration. This is even more important when users are not located, for example in remote assistance scenarios. Many approaches have been explored to allow users to share a point of view from their workspace, from mobile devices to robotic systems equipped with cameras and other types of devices. In this paper, we focus on augmented reality with HMD devices. Most HMDs are equipped with cameras and leave the user's hands free. Simply turning the head shows remote users the local workspace from a first-person perspective. However, this does not mean that remote users see exactly the same thing as the HMD wearer.

[0005] As illustrated in Figures 1 and 2, there can be a significant gap between the two. This is the case, for example, with Microsoft's Hololens 2 HMD: the center The FoV (Field of View) of the HMD cameras is higher than the center of the HMD wearer's FoV. This results in a shifted view for remote users, who only partially see the main object of interest.

[0006] This phenomenon can create confusion and hinder collaboration between users. While it may not have a significant impact in some application contexts, slowing down collaboration and creating misunderstandings can be very problematic in critical contexts such as industrial teleassistance or augmented surgery.

[0007] One solution could be to let remote users ask the HMD wearer to reorient their head in order to better see the scene. This could slow down collaboration and require remote users to guide the HMD wearer to a more appropriate viewpoint. Another approach could be to allow HMD wearers to visualize what remote users can see, so that they can adjust their FoV as needed. However, this approach places the burden of controlling and adapting the FoV on the HMD wearer. While acceptable for simple tasks, this additional load is not suitable for mentally demanding tasks such as surgery.

[0008] The following paragraphs summarize recent advances in augmented reality with an HMD for remote collaboration. We also examine attention guidance techniques and shared activity cues that could encourage a local user to adjust their field of view for remote users. View local workspace

[0009] There are a large number of studies on collaborative AR in many application areas, including medicine, remote assistance in industrial settings, project management and project management, or even education. But regardless of the application area, conversational cues such as voice and gestures are often not sufficient to enable effective remote collaboration. The system must also provide remote users with a way to perceive the local workspace. Some authors have explored the use of mobile devices to provide a first-person view of the scene to remote users. However, this approach requires local users to hold the device with one or both hands, which is not suitable for many contexts such as augmented surgery or manufacturing.

[0010] HMDs and other body-worn devices allow for sharing a first-person view of the workspace while leaving the wearer's hands free. For example, some authors have proposed a system in which a 360° camera is mounted on an HMD. Such a system allows the remote assistant to explore the entire of the scene panorama, even though the origin of the viewpoint is still linked to the local user's head. To avoid this problem, there is a system based on a rolling platform with cameras to allow a remote user to remotely explore the workspace. This system is nevertheless designed to offer an external and independent point of view of the scene. It can be difficult to obtain the exact first-person view of the local user (view "through the eyes") due to the size of the platform.

[0011] In the present invention, we want remote users to have the same view as the local user. Therefore, we focus on HMDs that stream the mixed reality scene in real time. Explicitly guide the HMD wearer

[0012] If remote users cannot see exactly what they want, they can simply guide the local user to a more appropriate viewpoint. Many previous studies have investigated guidance techniques for this type of scenario. For example, explicit guidance cues can be divided into four main categories: pointer-based techniques, sketches and annotations, hand gestures, and object model-based techniques. The first three categories are common, especially annotations. However, object model techniques are less common. This category includes studies in which a virtual model of the task objects is shared with the local user to illustrate complex steps. For example, some authors have proposed using virtual replicas of engine parts to facilitate assembly tasks.Beyond these four categories, another distinction can be made: the intention to 1) guide attention toward a specific target (e.g., an object) or 2) encourage the local user to adopt a given viewpoint on the scene. Many studies have examined AR techniques and visualizations to guide attention toward an object or location. Some techniques also focus on off-screen targets to compensate for the limited FoV of AR devices. On the contrary, some previous studies have worked on techniques to guide attention toward another viewpoint. For example, some authors have proposed an AR system with extendable virtual arms to help the HMD wearer position themselves correctly in the workspace. Other authors have proposed a visualization technique to guide a user toward a constrained set of positions and orientations.

[0013] Regardless of these categorizations, it is important to keep in mind the cognitive aspects related to explicit guidance techniques. For example, it has been observed that the direction of guidance can have an influence on the interaction and cognitive load of the participants. The influence of spatial reference on visual attention and workload was studied and revealed that egocentric perspective guidance can be more effective than allocentric guidance.

[0014] In the present invention, we assume that letting remote users explicitly guide the local user to adjust the FoV slows down collaboration and increases cognitive load. Instead, we wish to study automatic techniques, without intervention from remote users. Furthermore, we focus only on egocentric guidance. Implicit sharing of activity indices

[0015] In order to avoid explicit feedback from remote users, previous studies have explored implicit cues. Sharing activity cues can help the HMD wearer understand when to adjust the FoV to the current attention point of their collaborator. Many studies consider gaze as a promising implicit cue to communicate activity and intentions. For example, authors have investigated gaze sharing and the influence of the remote user's awareness of gaze sharing. Others have compared gaze and head gaze, while still others have also compared with a FoV cue. Although head-gaze is slower than eye-gaze, it can also be found to be more useful, easier to use, and less confusing. Other authors have explored the combination of modalities, with a system based on shared 2.5D gestures and gaze.

[0016] In the present invention, the implicit cues require the HMD wearer to monitor them to track the activity of remote users. We hypothesize that this may cause distractions and increase the cognitive load of the HMD wearer. Our goal is to offload this task from the system. In this way, we hope that the HMD wearer can better focus on their task and adjust the FoV only when needed instead of monitoring the activity of collaborators. Summary of the Invention

[0017] The present invention aims to overcome all or part of the drawbacks of the prior art set out above by proposing automatic head orientation techniques, known as AHO. AHO techniques are techniques for automatically guiding the orientation of the head in AR with an HMD. Their objective is to encourage a local HMD user to adjust the orientation of his head to allow remote users to have the appropriate FoV of the scene.

[0018] Instead of relying on user actions, the invention proposes an automated approach. AHO techniques are AR techniques designed to automatically encourage users wearing the HMD to adjust their head orientation. This way, remote users can correctly see the salient parts of the scene.

[0019] An automatic detection and guidance approach facilitates the work of the local user wearing the HMD. Automation has advantages over letting users fully manage field of view adjustments.

[0020] An automatic approach distracts users less. The level of visual intrusion and occlusion is acceptable to encourage the HMD wearer to quickly adjust the FoV without too much affecting the main task at hand.

[0021] One of the principles of the invention is the variation of the levels of visual occlusion and the anchoring of visual markers.

[0022] For this purpose, the subject of the invention is a method for guiding the orientation of the head of a local user wearing an immersive HMD headset in the context of remote collaboration in augmented reality, said immersive headset comprising a camera capturing a scene and transmitting the captured image to at least one remote user, said method being characterized in that it comprises a step of automatically displaying an interactive graphic element encouraging the local user to reorient his head, and therefore his headset, when a shift is detected between what said local user sees and what the remote users see; said shift being able to tolerate a margin of error defined in relation to an area of ​​interest in the observed scene;the graphic element evolving according to the head movements of the local user and disappearing when the offset disappears, in other words, when the scene observed by the remote users corresponds to the scene viewed by the local user. Presentation of the drawings;

[0023] The figures are given purely for illustrative purposes for a better understanding of the invention without limiting its scope. The various elements may be represented schematically and are not necessarily to scale. Throughout the figures, identical or equivalent elements bear the same numerical reference.

[0024] It is thus illustrated in:

[0025] [Fig. 1]: prior art.

[0026] [Fig. 2]: Example of field of view misalignment in a collaborative environment with an augmented reality HMD; a) overview of the complete workspace; b) view of the local user wearing the HMD; c) view of the remote user seeing the video stream from the headset. The field of view is vertically shifted, partially masking the augmented scene.

[0027] [Fig. 3]: Overview of the proposed AHO techniques. Example where the local user must realign the FoV to the right so that remote users can better see the scene. Detailed description of embodiments

[0028] It should be noted that certain technical elements well known to those skilled in the art are recalled here to avoid any insufficiency or ambiguity in the understanding of the present invention. Proposed techniques for AHO guidance

[0029] The main factor that influenced the design of AHO techniques is the trade-off between user attention and the urgency of visual field shift. We wanted the HMD wearer to be able to adjust the FoV quickly and accurately when needed. However, creating visual occlusion and attracting the user's attention could have a significant impact on the ongoing task. The level of intrusion, grouping visual occlusion and distraction, therefore became our first design axis. The second axis we considered is the difference in spatial reference of the guidance cues. On the one hand, visual cues anchored to the HMD's FoV (hereafter referred to as FoV-anchored) are always visible to the user but can quickly create visual clutter. On the other hand, world-anchored cues are spatially anchored in the scene, but may be outside the HMD's augmented field of view.

[0030] By combining our two axes, we designed three AHO techniques: Tar-getZone, HeadAvatar and Mask. These techniques are illustrated in [Fig. 3].

[0031] TargetZone: The objective of the TargetZone technique is to limit visual occlusion. When a visual field shift is necessary, an empty red square and a blue arrow appear in the center of the user's visual field (landmarks anchored in the visual field). The blue arrow indicates the direction of the visual field shift to be performed and points to a blue visor in the user's visual field, the scene (landmark anchored in the world). The user simply has to align the red square (as for a 2D location task) with the visor to reach a viewpoint suitable for distant users. Once this viewpoint is reached, the visual landmarks disappear.

[0032] HeadAvatar: this technique consists of displaying a virtual head model in front of the user (reference anchored in the visual field). The orientation of the head and a blue arrow indicate the movement to be performed to obtain an adequate orientation of the head. These two references are adapted in real time according to the user's movements. This technique does not include an index anchored in the word. Nevertheless, we hypothesized that the display of a head model could be more explicit and clearer than abstract clues like in TargetZone, which could help users despite a higher level of intrusion.

[0033] Mask: We designed the mask technique to study how we could force users to adjust the FoV as soon as possible. To do this, we mask almost the entire augmented visual field with an opaque orange mask when the technique is triggered. Only a circular area centered on the target viewpoint is visible (anchored landmark in the world). If the user is far from this viewpoint (area outside the HMD's FoV), a small circular area on the side is left visible to indicate in which direction the user should turn his head (FoV-anchored landmark). To avoid creating too much distraction, the mask appears gradually. Furthermore, we chose a dark orange color to find an acceptable compromise between opacity and brightness.

[0034] An important aspect of AHO techniques is their automatic triggering. Determining the current object of interest to users and whether the remote control users have an appropriate view of this object is a complex task. Performing this detection robustly and in real time is beyond the scope of the present invention. Nevertheless, it is possible to obtain an approximation by using activity indices such as the gaze of the HMD wearer. Furthermore, each HMD affected by the visual field misalignment has a fixed offset that can be calculated or at least approximated empirically.

[0035] A reference point anchored in the world is necessary to indicate precisely where to turn the head.

[0036] Overall, TargetZone was the AHO technique that achieved the best compromise between visual occlusion, speed, and accuracy. To achieve the latter goal, it appears that a majority of users prefer to see the exact position they need to reach when turning their head. We observed this despite the fact that 1) the range of head movement was limited in our study and 2) participants did not need a high level of accuracy to validate a FoV realignment (the hidden targets we used for head ray casting were of a comfortable size). The low sense of accuracy reported for HeadAvatar suggests that cues anchored in the FoV are less effective and preferred than cues anchored in the world to guide participants to a new viewpoint.

[0037] Automated techniques avoid forgetting FoV realignment.

[0038] The non-automated SharedCursor benchmark technique achieved good performance and a sense of accuracy while minimizing the level of intrusion. However, the latter aspect can be a double-edged sword since Half of the participants forgot to realign their FoV one or more times during the trials.

[0039] Passive cueing based on the remote user's implicit data may sometimes not be sufficient. If the local user's cognitive load is high enough, the visual field realignment may be forgotten. On the contrary, we did not observe forgetting with the three AHO techniques. Of course, these three techniques created more visual occlusions and made visual cues appear in the center of the FoV. It would be interesting to compare them with an alternative version of SharedCursor in which a visual or audio cue would be triggered to attract the user's attention when a FoV realignment is necessary. Overall, we therefore encourage designers not to rely solely on passive cues related to the remote users' implicit data.Automated techniques and momentary visual occlusion may be more effective in allowing remote users to fully see the salient part of the scene and facilitate cooperation.

[0040] Forcing an immediate realignment of the FoV may be acceptable.

[0041] The invention also suggests that it is possible to force an immediate realignment of the visual field. With the mask technique, all participants ended up realigning their visual field as quickly as possible. Of course, this approach forces users to interrupt their task and partially hides their environment. Nevertheless, despite the extreme visual occlusion, the constraint imposed by Mask was not always perceived negatively by participants. Not allowing a choice can sometimes be perceived as simpler. We hypothesize that simply encouraging a change in FoV may leave users more freedom, but also more cognitive load since they must determine for themselves the balance between their current task and the need to help distant users see the scene. Automation can therefore be applied at two stages: 1) the detection of the need for FoV realignment and 2) the immediate constraint to perform this realignment.Depending on the context, task and user profile, forcing an immediate realignment could therefore be considered an appropriate option.

[0042] It is apparent from the present description that certain non-essential elements may be modified, replaced or deleted without departing from the scope of the invention. [Fig. 1]

[0043]

[0045] [Fig. 2] vSsusi S:«S g£j.ï^ £ü>ys Mssi j 3½ig;h ^s^i: œ^iSSi^

Claims

10 Claims

1.

Images