Audio indicators of user attention in AR / VR environments

Abstract

A method for notifying a user of attention from another user in an augmented reality / virtual reality (AR / VR) system is provided. The method includes displaying a first image to a first user of the AR / VR system on a first electronic display, and upon detecting that the first user is gazing at a second user of the AR / VR system or an avatar of the second user in the first image, notifying the second user of the gaze of the first user by changing parameters of an audio communication channel from the first user to the second user so as to make the first user more audible to the second user.

Description

Technical Field

[0001] The present disclosure relates to visual displays and display systems, and in particular to wearable displays and methods for wearable displays.

[0002] background

[0003] Wearable displays can be used to present virtual images to users, or to augment real-world images with dynamic information, data, or virtual objects. Virtual reality (VR) or augmented reality (AR) scenes can be three-dimensional to enhance the experience and match virtual objects with real objects observed by the user. The user's eye position and gaze direction and / or orientation in space can be tracked in real time, and the displayed scene can be dynamically adjusted based on the user's head orientation and gaze direction to provide a better experience of being immersed in a simulated or augmented environment.

[0004] Wearable displays can be used to simulate or enhance social interactions between users. For example, in a virtual meeting application, users represented by avatars in a virtual meeting room can interact with each other by talking to each other, presenting information to a group of users, etc. One disadvantage of AR / VR simulations of social gathering events is that users may feel isolated or disoriented by the AR / VR environment, making social connections between users less direct.

[0005] Overview

[0006] The methods and systems disclosed herein can be used to enable a user to attract the attention of other users in an AR / VR environment. The AR / VR system may include a controller and multiple wearable displays worn by different users. These wearable displays may each include a visual display module, an eye tracking system, and an audio system, the visual display module is used to display VR images to the user and / or enhance the observed external world with AR images and signs, the eye tracking system is used to determine parameters such as the user's gaze direction, vergence, eye dilation, etc. in real time, and the audio system is used for audio communication with other users. The controller may include a remote or distributed computing system and a communication network that connects the controller to the wearable display, so that the wearable display is communicatively coupled to the controller. The term AR / VR used herein includes AR systems, VR systems, and any combination thereof, including mixed reality (MR) systems. Examples of AR / VR systems will be further given below.

[0007] A wearable display for a particular user may display images representing other users of the system. The method enables a user of an AR / VR system to notice when another user of the system is looking at them. The user represented on the screen may be located far away and invisible to the user wearing the particular wearable display, and may be represented by an avatar. In some AR embodiments, users may view other users directly through their AR headsets.

[0008] According to the present disclosure, a method in a wearable display of an augmented reality / virtual reality (AR / VR) system is provided. The method includes emitting audio to a second user of the AR / VR system using the wearable display, wherein the audio originates from a first user of the AR / VR system and has a parameter of an audio communication channel from the first user to the second user, wherein the parameter is changed from a first value to a second value so as to make the first user more audible to the second user. The parameter is changed when, for example, a processor executing instructions to process eye tracking data of the first user has detected a gaze of the first user directed to the second user or an avatar of the second user, and the avatar of the second user is in an image displayed to the first user.

[0009] The parameter may be changed from the first value to the second value when the processor executes the instructions to determine that the first user's eyes have been looking at the second user or the second user's avatar for at least a predefined time interval.

[0010] When the gaze direction of the first user at the second user has been detected, the parameter can be changed from the first value to the second value. In this case, when at least one of the gaze vergence of the first user at the second user or the pupil dilation of the first user has been detected, the parameter can be changed from the first value to the second value.

[0011] Prior to the first user's gaze, the parameter may have a first value.

[0012] When the processor determines at least one of a gaze direction or a gaze vergence of the first user at regular time intervals, the parameter may be changed from the first value to the second value.

[0013] When the gaze has been withdrawn for at least a predefined time interval, the parameter may be changed back to the first value.

[0014] According to the present disclosure, there is provided a non-transitory memory having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to emit audio to a second user of an AR / VR system using a wearable display, wherein the audio originates from a first user of the AR / VR system and has a parameter of an audio communication channel from the first user to the second user, wherein the parameter is changed from a first value to a second value to make the first user more audible to the second user. The parameter is changed when, for example, the processor executing instructions to process eye tracking data of the first user has detected a gaze of the first user directed toward a second user or an avatar of the second user, the avatar of the second user being in an image displayed to the first user.

[0015] The parameter may be changed from the first value to the second value when the processor executes the instructions to determine that the first user's eyes have been looking at the second user or the second user's avatar for at least a predefined time interval.

[0016] When the gaze direction of the first user at the second user has been detected, the parameter can be changed from the first value to the second value. In this case, when at least one of the gaze vergence of the first user at the second user or the pupil dilation of the first user has been detected, the parameter can be changed from the first value to the second value.

[0017] Prior to the first user's gaze, the parameter may have a first value.

[0018] When the processor determines at least one of a gaze direction or a gaze vergence of the first user at regular time intervals, the parameter may be changed from the first value to the second value.

[0019] When the gaze has been withdrawn for at least a predefined time interval, the parameter may be changed back to the first value.

[0020] According to the present disclosure, an AR / VR system is also provided, comprising: a first wearable display including an eye tracking system, a first electronic display for displaying a first image to a first user, and a microphone; a second wearable display including a second electronic display for displaying a second image to a second user and a speaker; and a controller communicatively coupled to the first wearable display and the second wearable display. The controller is configured to establish an audio communication channel from a first user of the AR / VR system to a second user, and to provide data for displaying the first image to the first wearable display. The controller includes an attention monitor configured to receive data related to eye detection and tracking of the first user from the first wearable display, and to identify a gaze of the first user directed to the second user or an avatar of the second user in the first image. The controller is configured to change a parameter of the audio communication channel from a first value to a second value once a gaze is detected, so as to make the first user easier to be heard by the second user.

[0021] The attention monitor may be configured to determine that the first user's eyes are looking at the second user or an avatar of the second user in the first image for at least a predefined time interval.

[0022] The attention monitor may be configured to determine a gaze direction of the first user. In this case, the attention monitor may be configured to determine at least one of a gaze vergence or a pupil dilation of the first user.

[0023] The parameter may include at least one of a peak audio frequency, a spectral width or a loudness of the audio communication channel.

[0024] The attention monitor may be configured to monitor the first user's gaze over time and, when the gaze has been withdrawn for at least a predefined time interval, change the parameter back to the first value. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Exemplary embodiments will now be described with reference to the accompanying drawings, in which:

[0027] Figure 1 is a block diagram of an AR / VR system according to the present disclosure;

[0028] Figure 2 yes Figure 1 A flowchart of a method in an AR / VR system;

[0029] Figure 3 It is shown Figure 1 a graph of a spectral response function of an audio communication channel;

[0030] Figure 4 is used for Figure 1 A flowchart of an exemplary method of an attention monitor for an AR / VR system;

[0031] Figure 5A is an isometric view of an embodiment of a near-eye augmented reality (AR) / virtual reality (VR) wearable display in a glasses form factor;

[0032] Figure 5B yes Figure 5A A side cross-sectional view of an AR / VR wearable display;

[0033] Fig. 6A is an isometric view of a head mounted display head mounted device of the present disclosure;

[0034] Figure 6B Yes Fig. 6A A block diagram of a virtual reality system of a head mounted device;

[0035] Figure 7 It is a schematic diagram of eyes staring at the screen;

[0036] Figure 8 According to the embodiment Figure 1 A block diagram of an example network environment for an AR / VR system; and

[0037] Fig. 9 is a block diagram of a computer system for implementing the methods of the present disclosure.

[0038] Detailed Description

[0039] Although this teaching is described in conjunction with various embodiments and examples, it is not intended that this teaching is limited to such embodiments. On the contrary, as will be appreciated by those skilled in the art, this teaching includes various alternatives and equivalents. All statements describing the principles, aspects and embodiments of the present disclosure and its specific examples herein are intended to include their structural and functional equivalents. In addition, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, that is, any element developed to perform the same function, without considering structure.

[0040] As used herein, unless expressly specified, the terms "first," "second," etc. are not intended to imply a sequential order, but rather to distinguish one element from another. Similarly, unless expressly specified, the sequential order of method steps does not imply a sequential order in which they are performed. The term "plurality" is used herein to mean "two or more," and the word "include" has the same meaning as the word "comprise."

[0041] refer to Figure 1, the AR / VR system 100 allows interaction between two or more users, such as users Ann (111) and Bob (112). The system 100 includes a controller 152, which is connected to two or more wearable displays 128 (Ann's wearable display) and 126 (Bob's wearable display) via a network. Each wearable display 126, 128 includes an electronic display and an audio system, that is, Ann's wearable display 128 includes an electronic display 121 and an audio system 191, and Bob's wearable display 126 includes an electronic display 122 and an audio system 192. The audio systems 191, 192 can each include at least one of a speaker 193 or a microphone 194. Images and sounds are transmitted using data received from the controller 152. The controller 152 also establishes a first audio communication channel 171 from Ann to Bob, and a second audio communication channel 172 from Bob to Ann. The first audio communication channel 171 and the second audio communication channel 172 can be part of a two-way communication channel between Ann and Bob, and / or a multi-way communication channel between Ann, Bob and other users (if any). In addition, the controller 152 can generate other sounds, such as soundtracks, background music, automatic notifications, etc.

[0042] The image generated by Ann's wearable display 128 may include an image captured by a wide-angle camera, or may include an image generated entirely by VR. The VR-generated image may represent multiple users, and in particular, may include an image 136 representing Bob. Herein, the image 136 representing Bob is referred to as Bob's avatar 136. Bob's avatar 136 may be a still or dynamic image, an icon, a graphical representation, an animated image, and the like. In some AR embodiments, Ann may look directly at Bob through an AR head-mounted device that transmits external light so that the outside world can be viewed directly, while enhancing the real view with computer-generated graphics and other computer-generated images. In such an embodiment, Bob's real-world view may be enhanced with prompts, signs, and the like. The same applies to Ann and any other user, as well as images representing other users to each other (i.e., avatars 131, 132, and 136 of other users). In some AR embodiments, virtual avatars and real people may be placed in the same scene. In addition, a user may dial in by phone and be placed in the scene by placing an avatar representing the dial-in user.

[0043] Figure 2is a flow chart of a method for presenting an audio prompt according to the present disclosure. The method includes establishing (225) an audio communication channel from a first user of an AR / VR system to a second user, and displaying (230) a first image to the first user of the AR / VR system on a first electronic display. Figure 1 In the example shown, the first user 111 is Ann, the first electronic display is her electronic display 121, and the image 136 represents Bob.

[0044] Further references Figure 1 , the first wearable display 128 of the AR / VR system 100 includes an eye tracking system 142, which collects data about the eyes of the first user and provides the obtained data to the attention monitor 161 (in Figure 1 Preferably, the second wearable display 126 also includes an eye tracking system so that Ann can also be notified of Bob's gaze.

[0045] The attention monitor 161 is a functional block that includes one or more local parts 161B within a wearable display (e.g., Ann's wearable display 128) and a central part 161A within the controller 152. Some functions can be implemented by the local part 161B at the user's location or by the central part 161A at a central location. The attention monitor 161 can be implemented with instructions stored in a computer-readable storage medium and executed by a hardware processor.

[0046] Attention monitor 161 synchronizes information obtained by eye tracking system 142 with information related to the image currently displayed on electronic display 121 to identify that Ann is looking at Bob (e.g., directly at Bob in an AR application, or at Bob's avatar 136 in a VR application). Figure 1 , line 170 indicates the specific direction of Ann's gaze as she looks at Bob's avatar 136 .

[0047] In this example, Ann is looking at Bob's avatar 136. Ann can shift her attention to image 132 representing another user. Attention monitor 161 can distinguish between images 136 and 132 based on the initial information used to form the image provided to electronic display 121. Attention monitor 161 has information identifying which portion of electronic display 121 is occupied by the image of a particular user. For AR applications, each AR head-mounted device can be equipped with a position sensor or position indicator, so that the AR / VR system 100 can determine the position of each user in 3D space in real time. This information, together with the gaze direction and / or convergence of various users in 3D space, enables the AR / VR system 100 to determine in real time for each user which or which other users may be looking at the particular user. For example, if Bob is at real-world location A, and Ann happens to be looking at the same real-world location A in 3D space, the AR / VR system 100 can determine that Ann is looking at Bob.

[0048] For the purpose of identifying the duration of a gaze of one user of the AR / VR system 100 directed at another user, the attention monitor 161 can calculate the duration of Ann's gaze at Bob or Bob's avatar 136. Gaze duration can be defined as an uninterrupted, continuous gaze at a specific image (e.g., Bob's avatar 136) or a specific person or 3D spatial location in an AR application during a predefined time interval. The gaze detection condition can be configured to view a specific image for at least a predefined time interval T1. When the gaze detection condition is met, the occurrence of a gaze is detected.

[0049] therefore, Figure 2 The method also includes detecting (235) a gaze, such as a gaze of a first user (Ann in this example) toward a second user (Bob) or an avatar of the second user (Bob's avatar 136) in a first image displayed to the first user (Ann) on the first electronic display 121, which is represented by line 170. Detecting 235 may include tracking eye movements of the first user by an eye tracking unit (e.g., eye tracking system 142). Detection may include a condition that the gaze duration is longer than a predefined threshold amount of time. Detection may also take into account some eye features, such as pupil dilation of the first user. A dilated pupil may indicate a high degree of focus. When the attention monitor 161 determines that a user is paying attention to another user for at least a predetermined threshold amount of time, the other user may be notified. For example, when the monitor 161 detects a gaze 170 from Ann toward an image 136 associated with Bob, the controller 152 may notify Bob.

[0050] The method also includes a notification 240. The notification 240 may include changing a parameter of the first audio communication channel 171 from the first user 111 (Ann) to the second user 112 (Bob) so as to make the first user 111 more audible to the second user 112. The parameter is changed from a first default value (e.g., a value common to all users) to a second different value. The parameter may include, for example, an audio signal level, i.e., the loudness of Ann's voice when Ann and Bob are talking. In this regard, this may facilitate social interaction between various users of the AR / VR system 100 by making it easier to attract the attention of Bob or any other user.

[0051] Based on gaze detection 235, any parameter that makes the first user's voice more audible or noticeable to the second user may be modified. For example, the parameter may be related to the spectral shape of the audio communication channel. Figure 3 And further reference Figure 1 , the first communication channel 171 ( Figure 1 ) has a default spectral response function of 300( Figure 3 ), the default spectral response function 300 may be common to all users of the AR / VR system 100. The default spectral response function 300 may be characterized by a first peak audio frequency 302 and a first spectral width 304. Upon detecting 235 the gaze of the first user 111 at the second user 112, the controller 152 changes the first communication channel 171 to have a modified spectral response function 310 characterized by a second higher peak audio frequency 312 and a second narrower spectral width 314. For example, when the first user 111 gazes at the second user 112 for a predetermined amount of time, the modified spectral response function 310 makes the sound sound sharper and, therefore, more easily noticed by the second user 112.

[0052] In some embodiments, frequency shifting techniques may be applied to change the pitch of a user's voice in real time, to change the modeled acoustic parameters of a user's throat in real time to make it sound different, and so on. Certain frequencies of the first user's voice may be blocked, and / or background sounds may be reduced to emphasize the first user's voice. In addition, in some embodiments, active (electronic) external noise suppression / noise cancellation may be employed, and when looking at a particular user, noise cancellation may be turned on or increased to achieve greater effect. The controller 152 may also be configured to suppress the voices of people next to the user, especially if these people are talking to each other rather than to the user.

[0053] The notification may also optionally include visual information added to the image of Ann displayed to Bob on Bob's display 122, such as a logo, visual cue, text, symbol, and / or some modification to Ann's avatar 131 (e.g., color, contrast, brightness, outline, etc.). The visual cue may represent an artificial image or feature provided by Bob's AR headset. Preferably, the visual cue is pre-defined and easily noticed, however the visual cue can reasonably remain subtle. The purpose of the visual cue is to facilitate drawing the attention of the second user to the first user in response to an identified gaze of the first user at or toward the second user.

[0054] The method can be performed in real time. When the user's eyes move between different areas of the screen, the AR / VR system 100 can receive updated eye tracking information at regular time intervals, and the display system can calculate the point of regard (i.e., gaze point) based on the updated eye tracking information, which is the point in the screen that the user is looking at. Preferably, the gaze point represented by the gaze direction and the gaze convergence is identified at predefined regular time intervals. As described above, as long as the gaze of the first user is reliably detected, that is, if the gaze has occurred for a predetermined amount of time, the second user is notified. Then, the system continues to monitor the gaze of the first user 111. The gaze withdrawal condition can be configured to continue looking at the second user for less than a predefined time interval T2. The two time intervals T1 (detection of the gaze threshold time) and T2 (withdrawal of gaze) can be equal, T1=T2, but they are not necessarily equal. The attention monitor 161 can also be configured to detect the user's natural eye and / or head movements so as to limit the duration of eye contact to a comfortable duration. In other words, the first user may briefly look at the floor, turn their head from side to side, etc. The attention monitor 161 may be configured to allow such movement within the "brief withdrawal time" T3 without changing the parameters of the audio communication channel between the first user and the second user. However, when the gaze withdrawal condition is met, the parameters of the first audio communication channel 171 are changed back to the original (first) value.

[0055] Eventually, the first user 111 may return their attention to the second user 112. When the gaze detection condition is met, the parameter of the first audio communication channel 171 may be changed to the second value again, or may be changed to a different third value selected, for example, to make the first user's voice more noticeable to the user being looked at. Herein, the second or third value of the audio parameter indicates attention, while the first (default) value may indicate a lack of attention.

[0056] From the perspective of the second user Bob (112), the method includes using Bob's wearable display to emit audio that attracts Bob's attention. The audio originates from Ann and has parameters of the audio communication channel 171 from Ann (first user 111) to Bob (second user 112). As explained above, the parameters are changed from a first value to a second value so that Ann (first user 111) is more easily heard by Bob (second user 112). This occurs when Ann's gaze directed at Bob or Bob's avatar 136 in an image displayed to Ann on Ann's electronic display 121 has been detected by a processor executing instructions to process Ann's eye tracking data. When the processor executes instructions to determine that the eyes of the first user 111 (Ann) have been looking at the second user 112 (Bob) or the second user's avatar 136 for at least a predefined time interval, the parameters can be changed from the first value to the second value. In some embodiments, when the gaze direction, vergence, pupil dilation, etc. of the first user at the second user 111 have been detected, the change of audio parameters can occur. At least one of a gaze direction or a gaze vergence of the first user may be determined by the processor at regular time intervals and may be changed back to the first value when the gaze has been withdrawn.

[0057] In some embodiments, facial recognition can be used to determine the presence of a person in a scene, as well as to predict the pitch of the person's voice based on age and gender. For VR applications, an audio profile can be provided for each user, and spectral filtering can be applied based on the audio profile to optimize voice detectability. In addition, in some applications, the auditory frequency response of each user can be determined by, for example, performing an auditory test using the user's own head-mounted device. As explained herein, the measured auditory frequency response can be taken into account when providing an audio signal to a user to enhance the user's voice detectability and / or highlight the audio communication channels of other users who are looking at that particular user.

[0058] Figure 41 is a flow chart of an exemplary method 400 employed by the attention monitor 161 for detecting the presence or absence of a gaze of the first user 111 toward the second user 112. Initially (402) the gaze state is no. An interval timer is started (404) and a gaze point is detected (406). If the gaze point is within the image of the second person (408), the look timer is incremented (410). If not, and if the interval timer is less than T1 (412), the process returns to detecting the gaze point (406). If no, the gaze state is determined (414). If there is no gaze and the look timer is less than X*T1 (416), the process restarts the interval timer (404). If the look timer is greater than X*T1 (418), the gaze state is yes (420) and the second user is notified of the gaze change (422). If the gaze continues and the look timer is greater than Y*T1, the process also restarts the interval timer (404). If the look timer is less than Y*T1, the gaze state is no (424), and the second user is notified of the gaze change (422). The gaze of the user can be continuously monitored over time. Here, X and Y are process parameters greater than 1. The possible gaze of the first user 111 towards other users represented on the first electronic display 121 can be considered in a similar manner. Of course, other methods can be used, in particular, a sliding time window can be used to determine the presence or absence of a gaze.

[0059] The first user 111 and the second user 112 may be human beings so that their gaze may be captured by an eye tracking device. However, the second user 112 notified of having the attention of the first user may also be a machine, a computer or a robot.

[0060] refer to Figure 5A and Figure 5B , the near-eye AR / VR display 500 is an embodiment of the wearable display 126, 128. As shown, the body or frame 502 of the near-eye AR / VR display 500 has the form factor of a pair of glasses. The display 504 includes a display component 506 ( Figure 5B ), display component 506 provides image light 508 to eyebox 510, which is a geometric area that can present a high-quality image to the user's eye 512. Display component 506 may include a separate near-eye AR / VR display module for each eye, or one AR / VR display module for both eyes.

[0061] The electronic display of the display assembly 506 may include, for example, but not limited to, a liquid crystal display (LCD), an organic light emitting display (OLED), an inorganic light emitting display (ILED), an active matrix organic light emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, a projector, or a combination thereof. A waveguide structure may be provided to transmit the image generated by the display to the user's eyes. In AR applications, a transparent or translucent waveguide structure may extend over the display assembly 506. The waveguide may include a selective reflector and a diffraction grating. The waveguide may also include a pupil replicating waveguide portion.

[0062] The near-eye AR / VR display 500 may also include an eye tracking system 514 for determining the gaze direction and / or vergence angle of the user's eye 512 in real time. Depending on the viewing angle and eye position, the determined gaze direction and gaze vergence angle can also be used for real-time compensation of visual artifacts. In addition, the determined vergence and gaze angle can be used for interaction with the user, highlighting objects, bringing objects to the foreground, dynamically creating additional objects or pointers, etc. The near-eye coherent AR / VR display 500 also includes an audio system 520. The audio system 520 may include a small speaker or headphones and / or a microphone.

[0063] Go to Fig. 6A , HMD 600 is an example of an AR / VR wearable display system that surrounds the user's face for greater immersion into an AR / VR environment. The function of HMD 600 is to enhance the view of the physical, real-world environment with computer-generated images, and / or to generate a completely virtual 3D image. HMD 600 includes a front body 602 and a strap 604. The front body 602 is configured to be placed in front of the user's eyes in a secure and comfortable manner, and the strap 604 can be stretched to secure the front body 602 on the user's head. The display system 680 can be arranged in the front body 602 for presenting AR / VR images to the user. The side 606 of the front body 602 can be opaque or transparent.

[0064] In some embodiments, the front body 602 includes a positioner 608, an inertial measurement unit (IMU) 610 for tracking the acceleration of the HMD 600, and a position sensor 612 for tracking the position of the HMD 600. The IMU 610 is an electronic device that generates data indicating the position of the HMD 600 based on measurement signals received from one or more position sensors 612, and the position sensor 612 generates one or more measurement signals in response to the movement of the HMD 600. Examples of the position sensor 612 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 610, or some combination thereof. The position sensor 612 can be located outside the IMU 610, inside the IMU 610, or some combination of these two locations.

[0065] The localizer 608 is tracked by an external imaging device of the virtual reality system so that the virtual reality system can track the position and orientation of the entire HMD 600. For improved tracking accuracy of the position and orientation of the HMD 600, the information generated by the IMU 610 and the position sensor 612 can be compared with the position and orientation obtained by tracking the localizer 608. As the user moves and turns in 3D space, accurate position and orientation are important for presenting an appropriate virtual scene to the user.

[0066] The HMD 600 may also include a depth camera assembly (DCA) 611 that captures data describing depth information of a local area surrounding part or all of the HMD 600. To this end, the DCA 611 may include a laser radar (LIDAR) or similar device. The depth information may be compared with information from the IMU 610 to more accurately determine the position and orientation of the HMD 600 in 3D space.

[0067] HMD 600 may also include an eye tracking system 614 for determining the orientation and position of the user's eyes in real time. Eye tracking system 614 may include an array of infrared illuminators for illuminating both eyes, a hot mirror for separating infrared light and visible light, and an eye tracking camera for obtaining binocular images through reflections (glints) from the illuminators. By comparing the position of the glint relative to the position of the pupil of the eye, the position and orientation of the eye can be determined. The position and orientation of the eye obtained also allow HMD 600 to determine the gaze direction of the user and adjust the image generated by the display system 680 accordingly. In one embodiment, vergence, i.e., the convergence angle of the gaze of the user's eyes, is determined. Depending on the viewing angle and the eye position, the determined gaze direction and gaze convergence angle can also be used for real-time compensation of visual artifacts. In addition, the determined vergence and gaze angle can be used for interaction with the user, highlighting objects, bringing objects to the foreground, creating additional objects or indicators, etc. An audio system 644 may be provided, which includes, for example, a group of small speakers and / or microphones built into the front body 602.

[0068] refer to Figure 6B The AR / VR system 650 includes Fig. 6A The HMD 600, the external console 690, and the input / output (I / O) interface 615, the external console 690 stores various AR / VR applications, setting and calibration programs, 3D videos, etc., and the input / output (I / O) interface 615 is used to operate the console 690 and / or interact with the AR / VR environment. The HMD 600 can be "tethered" to the console 690 with a physical cable, or via a communication interface such as A wireless communication link such as Wi-Fi is connected to the console 690. There may be multiple HMDs 600, each HMD 600 having an associated I / O interface 615, wherein each HMD 600 and I / O interface 615 communicate with the console 690. In alternative configurations, different and / or additional components may be included in the AR / VR system 650. In addition, in some embodiments, in combination with Fig. 6A and Figure 6B The functionality described by one or more components shown may be different in combination. Fig. 6A and Figure 6B The described manner is distributed among the components. For example, some or all functions of the console 690 may be provided by the HMD 600, and vice versa. The HMD 600 may be provided with a processing module capable of implementing such functions.

[0069] As referenced above Fig. 6AAs described, the HMD 600 may include an eye tracking system 614 (eg, for tracking eye position and orientation, determining gaze angle and convergence angle, etc.) Figure 6B ), an IMU 610 for determining the position and orientation of the HMD 600 in 3D space, a DCA 611 for capturing the external environment, a position sensor 612 for independently determining the position of the HMD 600, and a display system 680 for displaying AR / VR content to the user. The display system 680 includes ( Figure 6B ) electronic display 625, such as, but not limited to, a liquid crystal display (LCD), an organic light emitting display (OLED), an inorganic light emitting display (ILED), an active matrix organic light emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, a projector, or a combination thereof. The display system 680 also includes an optical block 630, whose function is to transmit the image generated by the electronic display 625 to the user's eyes. The display system 680 may also include a zoom module 635, which may be a part of the optical block 630. The function of the zoom module 635 is to adjust the focus of the optical block 630 to, for example, compensate for vergence-accommodation conflict, correct for visual defects of a specific user, offset aberrations of the optical block 630, etc.

[0070] The I / O interface 615 is a device that allows a user to send an action request and receive a response from the console 690. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end the capture of an image or video data, or an instruction to perform a specific action within an application. The I / O interface 615 may include one or more input devices, such as a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 690. The action request received by the I / O interface 615 is transmitted to the console 690, and the console 690 performs the action corresponding to the action request. In some embodiments, the I / O interface 615 includes an IMU that captures calibration data indicating the estimated position of the I / O interface 615 relative to the initial position of the I / O interface 615. In some embodiments, the I / O interface 615 can provide tactile feedback to the user according to the instruction received from the console 690. For example, tactile feedback may be provided when an action request is received, or when console 690 transmits instructions to I / O interface 615 that cause I / O interface 615 to generate tactile feedback when console 690 performs an action.

[0071] The console 690 may provide content to the HMD 600 for processing based on information received from one or more of the IMU 610, DCA 611, eye tracking system 614, and I / O interface 615. Figure 6B In the example shown, console 690 includes application storage 655, tracking module 660, and VR engine 665. Some embodiments of console 690 may have different Figure 6B Similarly, the functions described further below may be combined in different ways. Fig. 6A and Figure 6B The described approach is distributed among the components of console 690 .

[0072] The application storage 655 may store one or more applications for execution by the console 690. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be responsive to input received from the user via movement of the HMD 600 or the I / O interface 615. Examples of applications include: gaming applications, presentation and conferencing applications, video playback applications, or other suitable applications.

[0073] Tracking module 660 may calibrate AR / VR system 650 using one or more calibration parameters, and may adjust one or more calibration parameters to reduce errors in the determination of the position of HMD 600 or I / O interface 615. The calibration performed by tracking module 660 also takes into account information received from IMU 610 in HMD 600 and / or an IMU (if any) included in I / O interface 615. In addition, if tracking of HMD 600 is lost, tracking module 660 may recalibrate part or all of AR / VR system 650.

[0074] The tracking module 660 can track the movement of the HMD 600 or the I / O interface 615, the IMU 610, or some combination thereof. For example, the tracking module 660 can determine the location of a reference point of the HMD 600 in a map of the local area based on information from the HMD 600. The tracking module 660 can also determine the location of the reference point of the HMD 600 or the reference point of the I / O interface 615 using data indicating the location of the HMD 600 from the IMU 610 or using data indicating the location of the I / O interface 615 from an IMU included in the I / O interface 615, respectively. In addition, in some embodiments, the tracking module 660 can use the portion of the data indicating the location of the HMD 600 from the IMU 610 and the representation of the local area from the DCA 611 to predict the future positioning of the HMD 600. The tracking module 660 provides the VR engine 665 with the estimated or predicted future location of the HMD 600 or the I / O interface 615.

[0075] VR engine 665 may generate a 3D map of an area ("local area") surrounding part or all of HMD 600 based on information received from HMD 600. In some embodiments, VR engine 665 determines depth information for 3D mapping of the local area based on information received from DCA 611 regarding a technique used in calculating depth. In various embodiments, VR engine 665 may use the depth information to update a model of the local area and generate content based in part on the updated model.

[0076] The VR engine 665 executes an application within the AR / VR system 650 and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the HMD 600 from the tracking module 660. Based on the received information, the VR engine 665 determines the content to be provided to the HMD 600 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the VR engine 665 generates content for the HMD 600 that reflects (mirror) the user's movement in a virtual environment or in an environment that enhances a local area with additional content. In addition, the VR engine 665 executes an action within an application executed on the console 690 in response to an action request received from the I / O interface 615, and provides feedback to the user that the action is performed. The feedback provided may be visual or auditory feedback via the HMD 600 or tactile feedback via the I / O interface 615.

[0077] In some embodiments, based on the eye tracking information (e.g., the orientation of the user's eyes) received from the eye tracking system 614, the VR engine 665 determines the resolution of the content provided to the HMD 600 for presentation to the user on the electronic display 625. The VR engine 665 can provide content to the HMD 600 that has a maximum pixel resolution in the foveal region of the electronic display 625 where the user is gazing. The VR engine 665 can provide lower pixel resolution in other areas of the electronic display 625, thereby reducing the power consumption of the AR / VR system 650 and saving computing resources of the console 690 without compromising the user's visual experience. In some embodiments, the VR engine 665 can further use the eye tracking information to adjust the display position of objects on the electronic display 625 to prevent visual convergence and accommodation conflicts and / or to counteract optical distortion and aberrations.

[0078] Figure 7 The user's eye 702 is shown gazing at a screen 704 in a direction indicated by a gaze vector 706, which corresponds to Figure 1 The gaze vector 170. Figure 7 Also shown is a fovea area 708 on the screen. Attention monitor module 161 ( Figure 1 ) determines a gaze vector representing the direction the eye is looking. In some embodiments, the attention monitor module 161 determines the gaze vector based on a plurality of eye features associated with the eye, the plurality of eye features including the eye center ( Figure 7 ), the center of the cornea (A), the center of the cornea (C), the pupil (E), and the distance (h) between the center of the cornea and the center of the pupil. In one embodiment, the eye tracking system 142 estimates these eye features and sends these estimates as part of the eye tracking information to the attention monitor module 161. In another embodiment, the module 161 receives the angular orientation of the eye from the eye tracking system 142 and generates these eye features by applying a rotation to the eye model based on the angular rotation. In other embodiments, the attention monitor module 161 receives the foveal axis of the eye from the eye tracking system 142 and uses the direction of the foveal axis as the gaze vector 306. After determining the gaze vector 706, the module 161 determines the gaze point 714 by calculating the intersection between the gaze vector 706 and the screen 704. In other embodiments, the gaze point 714 is calculated by other means.

[0079] In another embodiment, two gaze vectors are determined for each of the user's two eyes, and the wearable display 128 determines the user's focus depth based on the convergence of the two gaze vectors. In this embodiment, the wearable display 128 determines the depth difference (hereinafter referred to as the depth difference) between the focus depth and the depth value of the object in the scene in order to determine whether the first user is focusing her eyes on a specific object or user, or just resting her eyes without paying attention to the screen.

[0080] Figure 8 An example of an AR / VR system 100 is provided in which the method of the present disclosure can be implemented. The AR / VR system 100 includes a network 855, one or more servers 850 including a controller 152, and multiple user devices (e.g., wearable displays 128 or 126) shown by devices 820 and 821. The server 850 provides data to the user device for displaying images, conveying user voices, monitoring user activities, and notifying the user of other users' attention to her. The network 855 provides a communication means between the server 850 and the user devices 820 and 821. The network 855 can be the Internet, a wireless or wired network such as a mobile device operator network, or any other network that can be used for communication between a server and a client. The user devices 820 and 821 can be HMDs, and each user device can include any suitable computing device, such as a desktop computer or a mobile device, such as a mobile phone, a cellular phone, a smart phone, a personal digital assistant, a notebook computer, a tablet computer, a laptop computer, a camera, a video camera, or a handheld game console.

[0081] Fig. 9 An example computer system 900 is shown. In certain embodiments, one or more computer systems 900 perform one or more steps of one or more methods described or shown herein. In certain embodiments, one or more computer systems 900 provide the functionality described or shown herein. In certain embodiments, software running on one or more computer systems 900 performs one or more steps of one or more methods described or shown herein, or provides the functionality described or shown herein. Certain embodiments include one or more portions of one or more computer systems 900.

[0082] The present disclosure contemplates any suitable number of computer systems 900. The present disclosure contemplates computer systems 900 that take any suitable physical form. By way of example and not limitation, computer system 900 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 900 may include one or more computer systems 900; may be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 900 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not as a limitation, one or more computer systems 900 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 900 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0083] In a particular embodiment, computer system 900 includes a processor 902, non-transitory memory 904, storage 906, an input / output (I / O) interface 908, a communication interface 910, and a bus 912. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0084] In a particular embodiment, the processor 902 includes hardware for executing instructions (e.g., those that constitute a computer program). As an example and not by way of limitation, to execute instructions, the processor 902 may retrieve (or fetch) instructions from an internal register, an internal cache, a non-transitory memory 904, or a storage device 906; decode these instructions and execute them; and then write one or more results to an internal register, an internal cache, a non-transitory memory 904, or a storage device 906. In a particular embodiment, the processor 902 may include one or more internal caches for data, instructions, or addresses. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0085] In a particular embodiment, the non-transitory memory 904 includes a main memory for storing instructions for execution by the processor 902 or data for operation by the processor 902. As an example and not as a limitation, the computer system 900 can load instructions from the storage device 906 or another source (such as, for example, another computer system 900) to the memory 904. The processor 902 can then load the instructions from the memory 904 to an internal register or an internal cache. In order to execute the instructions, the processor 902 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 902 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache. The processor 902 can then write one or more of these results to the non-transitory memory 904.

[0086] In a particular embodiment, the processor 902 executes only instructions in one or more internal registers or internal caches or in the memory 904 (rather than the storage device 906 or elsewhere), and operates only on data in one or more internal registers or internal caches or in the memory 904 (rather than the storage device 906 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 02 to the memory 904. As described below, the bus 912 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 902 and the memory 904 and facilitate access to the memory 904 requested by the processor 902. Although the present disclosure describes and illustrates a particular memory, the present disclosure contemplates any suitable memory.

[0087] In certain embodiments, storage device 906 includes a mass storage device for data or instructions. Storage device 906 may include removable or non-removable (i.e., fixed) media, where appropriate. Storage device 906 may be internal or external to computer system 900, where appropriate. In certain embodiments, storage device 906 is a non-volatile solid-state memory. Storage device 906 may include one or more storage devices 906, where appropriate. Although the present disclosure describes and illustrates specific storage devices, the present disclosure contemplates any suitable storage devices.

[0088] In a particular embodiment, the I / O interface 908 includes hardware, software, or both that provide one or more interfaces for communication between the computer system 900 and one or more I / O devices. Where appropriate, the computer system 900 may include one or more of these I / O devices. One or more of these I / O devices may implement communication between a person and the computer system 900. As an example and not as a limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device, or a combination of two or more of these devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 908 for them. Where appropriate, the I / O interface 908 may include one or more devices or software drivers that enable the processor 902 to drive one or more of these I / O devices. Where appropriate, the I / O interface 908 may include one or more I / O interfaces 908. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0089] In a particular embodiment, the communication interface 910 includes hardware, software, or both that provides one or more interfaces for communication (e.g., packet-based communication) between the computer system 900 and one or more other computer systems 900 or one or more networks. As an example and not by way of limitation, the communication interface 910 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (e.g., a Wi-Fi network). The present disclosure contemplates any suitable network and any suitable communication interface 910 for it. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0090] In particular embodiments, bus 912 includes hardware, software, or both that couples components of computer system 900 to each other. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0091] As used herein, references to computer-readable storage media include one or more non-transitory, tangible computer-readable storage media having a structure. As an example and not limitation, where appropriate, the computer-readable storage medium may include a semiconductor-based or other integrated circuit (IC) (e.g., a field programmable gate array (FPGA) or an application-specific IC (ASIC)), a hard disk, an HDD, a hybrid hard disk drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy disk, a floppy disk drive (FDD), a magnetic tape, a holographic storage medium, a solid-state drive (SSD), a RAM drive, a secure digital card, a secure digital drive, or another suitable computer-readable storage medium or a combination of two or more of these. As used herein, references to computer-readable storage media do not include any medium that is not eligible for patent protection under 35 U.S.C. §101. As used herein, references to computer-readable storage media do not include transient forms of signal transmission (e.g., propagating electrical or electromagnetic signals themselves), given that transient forms of signal transmission are not eligible for patent protection under 35 U.S.C. §101. Computer-readable non-transitory storage media may be volatile, nonvolatile, or a combination of volatile and nonvolatile, where appropriate.

[0092] The present disclosure contemplates one or more computer-readable storage media implementing any suitable storage. In a particular embodiment, where appropriate, the computer-readable storage medium implements one or more parts of the processor 902 (e.g., one or more internal registers or caches), one or more parts of the memory 904, one or more parts of the storage device 906, or a combination of these. In a particular embodiment, the computer-readable storage medium implements RAM or ROM. In a particular embodiment, the computer-readable storage medium implements volatile or persistent memory. In a particular embodiment, one or more computer-readable storage media embody software. In this article, where appropriate, references to software may include one or more applications, bytecodes, one or more computer programs, one or more executable files, one or more instructions, logic, machine code, one or more scripts or source code, and vice versa. In a particular embodiment, the software includes one or more application programming interfaces (APIs). The present disclosure contemplates any suitable software written or otherwise expressed in any suitable programming language or combination of programming languages. In a particular embodiment, the software is expressed as source code or object code.

[0093] The present disclosure is not limited in scope by the specific embodiments described herein. In fact, in addition to those embodiments and modifications described herein, various other embodiments and modifications will be apparent to those of ordinary skill in the art according to the foregoing description and accompanying drawings. Therefore, such other embodiments and modifications are defined as falling within the scope of the present disclosure. In addition, although the present disclosure is described herein in a specific environment for a specific purpose in the context of a specific implementation, a person of ordinary skill in the art will recognize that its usefulness is not limited thereto, and the present disclosure can be beneficially implemented in any number of environments for any number of purposes. Therefore, the claims set forth should be interpreted in light of the full breadth and spirit of the present disclosure as described herein.

Claims

1. A method in a wearable display of an AR / VR system, the method comprising: emitting audio to a second user of the AR / VR system using the wearable display, wherein the audio originates from a first user of the AR / VR system and has parameters of an audio communication channel from the first user to the second user, wherein when a processor executing instructions to process eye tracking data of the first user has determined that a gaze vergence of the first user is at the second user or at an avatar of the second user in an image displayed to the first user, the parameters are changed from a first value to a second value to make the first user more audible to the second user; wherein, before the gaze of the first user, the parameter has the first value; wherein when the processor determines the gaze vergence of the first user at regular time intervals, the parameter changes from the first value to the second value; and wherein the parameter is changed back to the first value when the gaze has been withdrawn for at least a predefined time interval.

2. The method according to claim 1, wherein: The parameter changes from the first value to the second value when the processor executes instructions to determine that the first user's eyes have been looking at the second user or an avatar of the second user for at least a predefined time interval.

3. The method according to claim 1, wherein: The parameter is changed from the first value to the second value when, during the gaze of the first user at the second user or at the avatar of the second user, a pupil dilation of the first user has been detected.

4. A non-transitory memory having instructions stored thereon, which when executed by one or more processors cause the one or more processors to emit audio to a second user of an AR / VR system using a wearable display, wherein: the audio originating from a first user of the AR / VR system and having parameters of an audio communication channel from the first user to the second user, wherein when a processor executing instructions to process eye tracking data of the first user has determined that a gaze vergence of the first user is at the second user or at an avatar of the second user in an image displayed to the first user, the parameters are changed from a first value to a second value to make the first user more audible to the second user; wherein, before the gaze of the first user, the parameter has the first value; wherein the parameter changes from the first value to the second value when the one or more processors determine the gaze vergence of the first user at regular time intervals; and wherein the parameter is changed back to the first value when the gaze has been withdrawn for at least a predefined time interval.

5. The non-transitory memory according to claim 4, wherein: The parameter changes from the first value to the second value when the processor executes instructions to determine that the first user's eyes have been looking at the second user or an avatar of the second user for at least a predefined time interval.

6. The non-transitory memory according to claim 4, wherein: The parameter is changed from the first value to the second value when, during the gaze of the first user at the second user or at the avatar of the second user, a pupil dilation of the first user has been detected.

7. An AR / VR system, comprising: a first wearable display comprising an eye tracking system, a first electronic display for displaying a first image to a first user, and a microphone; a second wearable display comprising a second electronic display and a speaker for displaying a second image to a second user; and a controller communicatively coupled to the first wearable display and the second wearable display and configured to establish an audio communication channel from the first user to the second user of the AR / VR system and provide data for displaying the first image to the first wearable display, the controller comprising an attention monitor configured to receive data related to eye tracking of the first user from the first wearable display and identify a gaze vergence of the first user at the second user or at an avatar of the second user in the first image, the controller configured to, upon determining the gaze vergence, change a parameter of the audio communication channel from a first value to a second value so as to make the first user more audible to the second user; wherein, before the gaze of the first user, the parameter has the first value; wherein the parameter is changed from the first value to the second value when the attention monitor determines the gaze vergence of the first user at regular time intervals; and wherein the parameter is changed back to the first value when the gaze has been withdrawn for at least a predefined time interval.

8. The AR / VR system according to claim 7, wherein: The attention monitor is configured to determine that the first user's eyes are looking at the second user or an avatar of the second user in the first image for at least a predefined time interval.

9. The AR / VR system according to claim 7, wherein: The attention monitor is configured to determine pupil dilation of the first user.

10. The AR / VR system according to claim 7, wherein: The parameter comprises at least one of a peak audio frequency, a spectral width or a loudness of the audio communication channel.

Images