Real-time iris detection and enhancement

Abstract

A realistic eye reflection is created for the virtual representation of the subject based on the lighting conditions defined by the environmental map. For each eye of a subject, a set of markers associated with feature points of the eye of the subject is tracked. A region corresponding to the opening of the eye is determined based on the marker. Within the area corresponding to the opening of the eye, an iris area is identified. A lighting effect is applied to the iris portion of the eye. An environment map defining illumination of a particular environment may be used to adjust the appearance of the iris region of the eye. The brightness in the iris region may be adjusted to cause the eye to have low light corresponding to illumination in the environment, thereby making the appearance of the eye more realistic.

Description

Background Art

[0001] A computerized character that represents and is controlled by a user is often referred to as an avatar. Avatars can take many forms, including virtual humans, animals, and plant life. Some computer products include avatars with facial expressions driven by the user's facial expressions. One use for face-based avatars is communication, where a camera and microphone in a first device transmits the first user's audio and a real-time 2D or 3D avatar to one or more second users, such as other mobile devices, desktop computers, video conferencing systems, and the like. Eyes are one of the most expressive and important features of the human face, and they convey a great deal of information about a person's emotions, intentions, and attention. Therefore, creating realistic eyes for an avatar can enhance immersion and interaction in a virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 An example flow diagram of a technique for rendering a virtual representation of a subject's eye is shown in accordance with one or more embodiments.

[0003] Figure 2 An example flow diagram illustrating a technique for applying lighting effects to an iris region of a virtual representation of a subject's eye is shown in accordance with one or more embodiments.

[0004] Figure 3 A technique for generating a target texture according to one or more embodiments is shown in flowchart form.

[0005] Figure 4 A diagram of a head mounted device is shown according to one or more embodiments.

[0006] Figure 5 A flow diagram illustrating a technique for rendering characters in a multi-user communication session is shown in accordance with one or more embodiments.

[0007] Figure 6 A multifunctional electronic device according to one or more embodiments is shown in block diagram form.

[0008] Figure 7 A computer system according to one or more embodiments is shown in block diagram form. DETAILED DESCRIPTION

[0009] The present disclosure generally relates to image processing. More particularly, but not by way of limitation, the present disclosure relates to techniques and systems for generating and rendering avatars using machine learning.

[0010] Embodiments described herein relate to creating realistic eye reflections in a virtual character based on lighting conditions defined by an environment map. For each eye of a subject, a set of markers is tracked. The set of markers can be associated with landmarks on the subject's eyes, such as the corners of the eyes and the top and bottom positions of the eye openings. Based on the markers, an area corresponding to the eye openings is determined, for example using polynomial interpolation. Within the area corresponding to the eye openings, an iris region can be identified, for example based on color differences within the eye openings. Lighting effects are applied to the iris portion of the eye. For example, an environment map defining the lighting of a particular environment can be used to adjust the appearance of the iris region of the eye. For example, the brightness in the iris region can be adjusted so that the eye has a shimmer corresponding to the lighting in the environment, thereby making the eye appear more realistic. Additionally or alternatively, visual artifacts can be applied to the eye to introduce shimmer or other features to make the eye appear more realistic.

[0011] Utilizing the embodiments described herein to render a character's eye region has numerous technical benefits. One benefit of using markers to interpolate the eye's position rather than using a machine learning model for eye and iris segmentation is reduced computational complexity and latency in the eye tracking system. Markers provide a simple and robust way to estimate the shape of the eye region based on the relative distance and angle between the marker and the camera. Additionally, applying lighting effects to the identified portion of the eye provides an efficient way to produce a realistic virtual representation of the subject while conserving computational resources.

[0012] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features, such as physical surfaces or physical objects. For example, a physical environment may correspond to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner that conforms to at least one law of physics. As an example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, an XR system can detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment and, in response, adjust the graphical content and sound field presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust characteristics of graphical content in an XR environment in response to an indication of physical movement (e.g., a voice command).

[0013] For the purposes of this disclosure, an autoencoder refers to a type of artificial neural network used to fit data in an unsupervised manner. The goal of an autoencoder is to learn an optimized representation of a set of data. An autoencoder is designed to reproduce its input values ​​as output while passing through an information bottleneck that allows the dataset to be described using a set of latent variables. This set of latent variables is a compressed representation of the input from which a decoder can generate output. A trained autoencoder will have an encoder portion, a decoder portion, and the latent variables represent an optimized representation of the data.

[0014] For purposes of this disclosure, the term "character" refers to a photorealistic virtual representation of a real-world subject such as a person, animal, plant, object, etc. A real-world subject may have a static shape, or may have a shape that changes in response to movement or stimulation.

[0015] Figure 1 An example flow chart of a technique for rendering a virtual representation of a subject's eyes is shown, according to one or more embodiments. Specifically, the flow chart shows an exemplary series of steps for applying lighting effects to a character's eyes, along with example illustrations of each step in the process. Figure 1The various processes shown are shown in a particular order, but it should be understood that the various processes described can be performed in a different order. In addition, it may not be necessary to perform all of the various processes to train the mesh and texture encoders and decoders or obtain the lighting representation.

[0016] Flowchart 100 begins at block 110 where tracking data for a face is obtained. According to one or more embodiments, a local device such as a head mounted device 114 may run a facial tracking algorithm. Facial tracking techniques may include, for example, applying image data and / or other sensor data captured by the local device 114 to obtain information about a posture or other characteristics of the user 112. In some embodiments, facial tracking techniques may include applying the sensor data to an expression model to obtain a set of expression latent variables for the user's expression in a particular frame. As another example, for each frame of tracking data, a height field displacement map may be generated and the height field displacement map provides RGB values ​​as well as depth values ​​and alpha values. The displacement map may include additional information, such as the location of certain features of the user in the form of semantic markers. Other information determined from the tracking data may include, for example, the location and orientation of the head, the position of the head, and the like.

[0017] Additionally, in some embodiments, eye tracking may be performed. According to one or more embodiments, the tracking data may additionally include eye tracking data. In some embodiments, the eye tracking process may include capturing image data and / or other sensor data for each eye to determine characteristics of the eye. Characteristics determined from the tracking data may include, for example, gaze direction, eye position, etc.

[0018] The flowchart 100 continues to block 120 where a shape or geometry corresponding to an eye region is determined in the tracking data. According to one or more embodiments, the eye region may include a region 124 of the image data 122 that includes the opening of one eye or the opening of the eye. For purposes of this disclosure, the term "eye region" may include a region for one or both eyes and may include a continuous region or two separate sub-regions, each corresponding to one of the eyes. The eye region may be determined in a variety of ways. For example, semantic segmentation may be used to identify a region of the image data captured from the tracking data that includes the opening of the eye. As another example, as will be described in Figure 2 As described in , feature detection can be used to identify feature points, such as semantic landmarks provided by tracking data, to determine regions of the image data corresponding to eye openings.

[0019] Flowchart 100 continues to block 130 where a viewing direction 132 of user 112 is determined. According to one or more embodiments, the viewing direction may be determined from the tracking data captured at block 110. For example, viewing direction 132 may be determined based on eye tracking data. For example, the user's eyes may be tracked by one or more eye tracking sensors to determine a gaze vector. Additionally or alternatively, the user's viewing direction may be determined based on device sensors within device 114, such as an accelerometer, a gyroscope, and the like. For example, if the device is a head-mounted device, the posture of the device may be considered a surrogate for user 112. Alternatively, posture information for device 114 may be used to supplement other posture data to determine the user's posture.

[0020] At block 140, a lighting map 144 for the environment is obtained. An environment map may be associated with a scene having specific lighting. Lighting map 144 may represent brightness, color, and / or other characteristics related to the lighting in the scene and may be any type of digital representation of the environment's lighting. According to some embodiments, lighting map 144 may be generated during runtime or may be predefined. Furthermore, lighting map 144 may represent the specific environment in which the subject exists, the specific environment in which the subject's representation is to be presented, or some combination thereof. For example, device 114 may include sensors such as a camera or an ambient light sensor that can capture sensor data to determine the brightness distribution of light sources in the environment. In some embodiments, the lighting map may be a dynamic lighting map that reflects the lighting in the environment based on the placement of virtual content that emits light. For example, the lighting map may include lighting characteristics from a user interface window, a virtual light emitter, or other component that may affect the lighting of the environment in which the user's representation is presented. Furthermore, the lighting map may alternatively or additionally be based on the lighting characteristics of the real-world environment in which the user's representation is to be presented, such as the lighting characteristics of a receiving device in which another person is viewing the user's virtual representation.

[0021] In some embodiments, the device 114 can be an enclosed device, such as a head-mounted device with a visor. To this end, a lighting map can be generated based on the lighting within the head-mounted device. Figure 4 , shows an example diagram according to one or more embodiments. A front view of a headset is shown as headset 400, and the relative positioning of headset 400 and a user's left eye 435L and right eye 435R. Additionally, in some embodiments, the lighting map can be based on the lighting characteristics of multiple environments. For example, the lighting within the headset can be combined with the lighting characteristics of additional environments, such as a physical or virtual environment in which a virtual representation of the user is to be presented.

[0022] Headset 400 may include a number of components that affect the illumination of eyes 435L and 435R. Headset 400 may include a left optical module 415L and a right optical module 415R. Left optical module 415L may include a left display 420L. Similarly, headset 400 may include a right optical module 415R. Right optical module 415R may include a right display 420R. When worn by a user, light from left display 420L may bounce off left eye 435L, while light from right display 420R may bounce off right eye 435R. Additionally, device 400 may include one or more sensors configured to capture sensor data from the user's eyes. As shown, an example device is a set of emitters 405A and 405B configured to emit light toward the eyes for eye tracking. For example, emitters 405A and 405B may emit light to illuminate the eyes. The image of the eye can then be captured by cameras 410A and 410B for eye tracking.Thus, when worn by a user, the light from emitters 405A and 405B can contribute to a lighting map of the environment within device 400.

[0023] At block 150, lighting effects are applied to the eye region based on the lighting map. In some embodiments, the environment map is reflected onto the eye region. For example, a determination can be made based on head pose which portion of the lighting map (i.e., the target area) should be reflected back onto the eye region, as shown in eye region 154 of image data 152. Additionally, in some embodiments, gaze information can be considered when determining the particular viewing direction of the eye. According to some embodiments, lighting effects can be applied based on the target area of ​​the lighting map, rather than reflecting the lighting map back onto the eye region. For example, the brightness for a corresponding area on the eye can be adjusted based on the target area of ​​the lighting map.

[0024] Additionally, in some embodiments, lighting effects may be applied only to specific portions of the eye, such as the iris area. Figure 2 An example flow chart illustrating a technique for applying a lighting effect to an iris region of a virtual representation of a subject's eye is shown in accordance with one or more embodiments. Figure 2 The various processes and diagrams shown are shown in a particular order, but it should be understood that the various processes described can be performed in a different order. Additionally, it may not be necessary to perform all of the various processes to train the mesh and texture encoders and decoders or to obtain the lighting representation.

[0025] Flowchart 200 begins at block 210 where image data of a subject's eyes is obtained. According to one or more embodiments, image data may be captured for each eye, such as left eye 212L and right eye 212R. In some embodiments, the image data may be captured by one or more cameras facing the user. For example, returning to Figure 4 , a set of cameras including camera 410A and camera 410B can be located in the head mounted device to capture images of the user's eyes. Thus, in some embodiments, image data can be captured from a single camera or from multiple cameras.

[0026] Return to Figure 2 Flowchart 200 continues to block 220, where eye region markers are identified. As described above, in some embodiments, eye region markers may be obtained as part of the tracking data and may identify feature points on the user's face corresponding to the opening of an eye or the outline of an eye opening. In the current example, markers 222A, 222B, 222C, and 222D correspond to markers for left eye 212L. Specifically, marker 222A indicates the location of the top of the eye opening. Marker 222B indicates the inner corner of the eye opening. Marker 222C indicates the lower edge of the eye position. Marker 222D indicates the outer corner of the opening for left eye 212L. Similarly, markers 222E, 222F, 222G, and 222H correspond to markers for right eye 212R. Specifically, marker 222E indicates the location of the top of the eye opening. Marker 222F indicates the inner corner of the eye opening. Marker 222G indicates the lower edge of the eye position. Marker 222H indicates the outer corner of the opening for right eye 212R.

[0027] In some embodiments, a feature identification process may be used to obtain markers in image data containing eyes. Additionally or alternatively, markers may be obtained from a process for capturing features of a user's face to be used to generate a virtual representation of a character or user. For example, for each frame of sensor data from an image of a subject, a height field displacement map corresponding to the face may be generated, and the height field displacement map provides RGB values ​​as well as depth values ​​and alpha values ​​based on which the features of the user may be converted into a character representation. According to some embodiments, the depth map may include additional information, such as the location of certain features of the user in the form of semantic markers. Thus, the semantic markers from the depth map may be converted into image data to identify the location of the markers in the captured image data. Although four markers are shown for each eye, alternative embodiments include obtaining a different number of markers for each eye.

[0028] Flowchart 200 continues to block 230, where eye regions are identified from eye region markers. According to one or more embodiments, a region for each eye may be determined based on the markers. Each marker in the markers may be associated with position information. The position information may be a 2D position corresponding to the displayed position of the eye, or a 3D position representing the physical location of a feature associated with a feature point in the physical environment. According to one or more embodiments, interpolation techniques may be applied to the positions of the markers to determine the boundaries of the eye regions. As shown in the example illustration, a left eye region 232L and a right eye region 232R are determined. The shapes of the left eye region 232L and the right eye region 232R are determined based on the interpolation techniques applied to the corresponding markers. According to some embodiments, errors may occur during interpolation, which may be compensated for by shrinking the boundaries inward toward the eye openings.

[0029] At block 240, an iris region is determined based on the color difference within the eye region. According to one or more embodiments, the color difference of pixels within the eye region in the image data may be analyzed. In some embodiments, the color difference between two groups of pixels can be used to identify the iris region within the eye region. That is, because the color of the iris differs from the color of the sclera, the color difference within the eye region can be used to identify the iris. As shown, the left eye region 232L may include an iris region 244L and the remainder of the eye region corresponding to the sclera region 242L. Similarly, the right eye region 232R may include an iris region 244R and the remainder of the eye region corresponding to the sclera region 242R.

[0030] The flowchart ends at block 250, where a lighting effect is applied to the iris region of the virtual representation of the subject based on the environment map. Figure 1 As described above, lighting effects can be applied by determining the subject's head pose or gaze and reflecting the environment map onto the eye area accordingly. For example, a determination can be made based on the head pose which portion of the lighting map (i.e., the target area) should be reflected back onto the eye area. According to some embodiments, lighting effects can be applied based on the target area of ​​the lighting map, rather than reflecting the lighting map back onto the eye area. For example, the brightness of the corresponding area on the eye can be adjusted based on the target area of ​​the lighting map.

[0031] In some embodiments, lighting effects may be applied only to the iris region of the virtual representation of the eye. For example, brightness may be applied only to the iris region according to a lighting map. Additionally, in some embodiments, lighting effects may be applied to both the sclera region and the iris region of the virtual representation of the eye. For example, lighting effects may be applied differently to the sclera region and the iris region of the virtual representation of the eye. For example, brightness may be applied according to a lighting map with a greater intensity in the iris region than in the sclera region. As shown in the example illustration, a virtual representation of the eye region 256 is shown that includes lighting treatment applied to the left iris region 254L and the right iris region 254R, but not to the left sclera region 252L and the right sclera region 252R.

[0032] As described above, generation of a virtual representation of an iris may be performed as part of a technique for generating a character of a subject. Figure 3 A technique for generating a target texture according to one or more embodiments is shown in flowchart form. The specific components and processes of the flowchart are intended to provide example embodiments and are not intended to limit the present application.

[0033] The flowchart begins at 302, where an enrollment image is received. The enrollment data may include, for example, image data of a user, which may be used to generate a character for the user. The enrollment data may be captured from a enrollment process during which the device captures sensor data of the user performing one or more expressions. The enrollment data may be captured from the same sensors used to track the user's expressions during runtime, or may be captured by different sensors. For example, in some embodiments, the enrollment process is performed by the user holding the device in front of them, such that the enrollment image is captured by a camera (such as a scene camera) that is generally facing away from the user when the device is worn. The enrollment image may include a general image of the face and / or the eye region, such that a virtual representation of the eye region may be generated during runtime.

[0034] During runtime of the communication session, expressive image data 304 is obtained. For example, image data 314 of the user's face may be captured for each frame during the communication session. In some embodiments, additional eye image data 316 may be captured. At 318, a mesh is generated based on the image data, for example, based on a trained network. The resulting mesh 320 may be applied to an expression model to obtain a set of expression latent variables 322. The set of expression latent variables provides a compact representation of the geometry of the user's face for a particular expression for a given frame. In some embodiments, the expression latent variables 322 may be used alone or in combination with the enrollment data 302 to generate a facial texture model 330. The facial texture model may provide a virtual representation of the character's texture based on the current expression image.

[0035] According to some embodiments, the eye model 324 can be trained to identify the eye region and optionally the iris region within the eye region. In this way, the eye model 324 can use the eye image data 316 to determine the eye region and / or iris region. In some embodiments, feature detection or semantic segmentation can be used to determine the eye region and / or iris region. Alternatively, the eye region and / or iris region can be determined by obtaining a marker of a feature point indicating the opening of the eye, from which the eye opening can be identified. According to some embodiments, the application of the eye model can be part of facial tracking, as described above with respect to Figure 1 Alternatively, the application of the eye model may be a separate process.

[0036] According to one or more embodiments, an environment map 310 may be provided. As described above, the environment map may correspond to a lighting map for the physical environment in which the subject is located, the physical or virtual environment in which the subject's character is to be presented, or the like. For example, the lighting map may provide a representation of the light within the environment. An eye shader 326 may be used to apply lighting processing to the eye region or iris region identified by the eye model 324. Lighting effects may be applied by determining the subject's head pose or gaze and reflecting the environment map onto the eye region accordingly. For example, a determination may be made based on the head pose which portion of the lighting map (i.e., the target area) should be reflected back onto the eye region. According to some embodiments, lighting effects may be applied based on the target area of ​​the lighting map, rather than reflecting the lighting map back onto the eye region. For example, the brightness for the corresponding area on the eye may be adjusted based on the target area of ​​the lighting map.

[0037] In some embodiments, lighting effects may be applied only to the iris region of the virtual representation of the eye. For example, brightness may be applied only to the iris region according to a lighting map. Additionally, in some embodiments, lighting effects may be applied differently to the sclera region and the iris region of the virtual representation of the eye. For example, brightness may be applied according to a lighting map with a greater intensity in the iris region than in the sclera region.

[0038] According to one or more embodiments, GPU shader 332 can generate a target texture 334 by combining eye texture 328 with generated facial texture model 330. For example, a luminance process for the eye texture can be applied by GPU shader 332 to the detected iris region. As another example, the eye region or the iris region can be generated as a separate texture and combined with the facial texture by GPU shader 332. Target texture 334 can then be used to render a character or other virtual representation of a subject by combining the texture with a geometric representation of the subject.

[0039] The embodiments described herein may be used during a multi-user communication session in which users on separate devices communicate with each other.During a communication session, remote users may be represented by roles. Figure 5 A flowchart illustrating a technique for rendering characters in a multi-user communication session, according to one or more embodiments, is provided. While various processes are described as being performed by specific devices, it should be understood that, in alternative embodiments, one or more of the processes may be performed by additional and / or alternative devices. Furthermore, the performance profiles of the various processes may vary. For example, some processes illustrated as being performed by a sending device may be performed by a receiving device.

[0040] The flowchart begins at 512, where device A 505 may capture image data for each frame during the communication session. The image data may include one or more images captured by a camera oriented toward the user, and specifically toward the user's eyes. The image data may include one or more images capturing all and / or part of the user's face. At 514, device A 505 may execute tracking algorithms, including, for example, face tracking and eye tracking. Facial tracking and eye tracking functionality may include, for example, applying the captured image data to various models to obtain information about the user. In some embodiments, the tracking algorithms may be used to determine the pose of the user's head in a particular environment. As described above, an eye network may provide eye tracking information, such as gaze direction. In some embodiments, the eye network may be part of facial tracking. That is, the facial tracking network may include the functionality of the eye tracking network, rather than executing a separate eye network. Alternatively, facial tracking and eye tracking may be implemented as separate processes.

[0041] Based on the tracking information, the process continues at block 516 and character geometry is generated by device A 505. According to one or more embodiments, the character geometry can be generated based on the tracking data and, optionally, based on registration data for a particular user. The geometry can be in the form of a 3D mesh, point cloud, or other three-dimensional representation of the user's shape.

[0042] At box 520, device A 505 generates a facial texture. The facial texture can be a representation of the appearance of the face. The facial texture can be generated in a variety of ways. In some embodiments, the facial texture is generated. In some embodiments, the geometry and texture can be generated in a single representation. For example, a representation of the user can be generated in the form of a depth map. According to some embodiments, the depth map may include a height field displacement map. The height field displacement map may be based on an RGBDA image (e.g., a red-green-blue-depth-alpha image). However, in some embodiments, unlike traditional depth images (i.e., RGBDA images) that define the depth of content relative to a single camera position, depth can be defined as a portion of the face relative to multiple points on the surface of a planar shape (such as a cylinder, a partial cylinder, or other curved 3D shape configured to cover the face of the subject).

[0043] According to one or more embodiments, device B 510 may receive virtual representation data, such as geometry, facial texture, pose, and an indication of the eye region, from device A 505, as shown at block 524, to generate the character. The indication of the eye region may include, for example, the coordinates of the geometry and / or texture where the eye and / or iris are located. At block 526, device B 510 obtains a lighting map. According to one or more embodiments, the lighting map may be obtained from device A 505, generated locally by device B 510, or obtained from another source. According to one or more embodiments, the lighting map may be associated with a scene having specific lighting. The lighting map may represent brightness, color, and / or other characteristics associated with the lighting in the scene and may be any type of digital representation of the lighting of the environment. According to some embodiments, the lighting map may be generated by device A 505 and / or device B 510 during runtime, or may be predefined, for example, if the character is to be rendered in a virtual environment. In addition, in some embodiments, multiple lighting maps may be obtained. This could happen, for example, if a user has local lighting from the local environment affecting the appearance of their eyes, and is being rendered in a secondary environment with different lighting that affects the appearance of the eyes. For example, by taking into account multiple lighting maps, characters can be rendered to appear as if they have local lighting combined with remote lighting.

[0044] At block 528, the character is rendered based on the geometry, texture, head pose, eye region, and environment map. Specifically, as described above, the character may be generated using geometry and texture. Lighting processing may be applied to the portion including the eye region to adjust the brightness of the character's eye or iris based on the lighting map. The final character may then be rendered at device B 510.

[0045] Figure 6A multifunctional electronic device according to one or more embodiments is shown in block diagram form. A simplified block diagram of a client device 675A communicatively connected to a client device 675B according to one or more embodiments of the present disclosure is shown. Client device 675A and client device 675B can each be part of a multifunctional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, base station, laptop computer, desktop computer, network device, or any other electronic device. Network device 675A can be connected to client device 675B across network 605. Exemplary networks include, but are not limited to, local networks (such as universal serial bus (USB) networks), local area networks of organizations, and wide area networks (such as the Internet). According to one or more embodiments, client device 675A and client device 675B can participate in a communication session in which each device can render an avatar of a user of the other client device.

[0046] Each of client device 675A and client device 675B may include a processor 682A and 682B, such as a central processing unit (CPU). Processor 682A and processor 682B may each be a system on a chip (such as those present in mobile devices) and may include one or more dedicated graphics processing units (GPUs). In addition, each of processor 682A and processor 682B may include multiple processors of the same or different types. Each of client device 675A and client device 675B may also include memory 684A and 684B. Each of memory 684A and memory 684B may include one or more different types of memory that can be used to perform device functions in conjunction with processor 682A and 682B. For example, each of memory 684A and memory 684B may include a cache, ROM, RAM, or any type of transient or non-transient computer-readable storage medium capable of storing computer-readable code. Each of memory 684A and memory 684B can store various programming modules for execution by processors 682A and 682B, including role modules 686A and 686B. Each of client device 675A and client device 675B can also include storage devices 618A and 618B. Each of storage devices 618A and 618B can include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). Each of storage devices 618A and 618B can include registration data 620A and 620B and model repositories 622A and 622B.

[0047] Each of client device 675A and client device 675B may also include one or more cameras 676A and 676B or other sensors, such as depth sensor 678A and depth sensor 678B, from which the depth of the scene can be determined. In one or more embodiments, each of the one or more cameras 676A and 676B can be a traditional RGB camera or a depth camera. In addition, each of the one or more cameras 676A and 676B may include a stereo or other multi-camera system, a time-of-flight camera system, etc., which captures images from which the depth information of the scene can be determined. Each of client device 675A and client device 675B can allow a user to interact with an extended reality (XR) environment. There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to a person's eyes. Display devices 680A and 680B can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology that projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, for example, as a hologram or on a physical surface.

[0048] Now refer to Figure 7, shows a simplified functional block diagram of an exemplary multifunction electronic device 700 according to one embodiment. Each of the electronic devices may be a multifunction electronic device, or may have some or all of the components described herein for the multifunction electronic devices. The multifunction electronic device 700 may include a processor 705, a display 710, a user interface 715, graphics hardware 720, device sensors 725 (e.g., a proximity sensor / ambient light sensor, an accelerometer, and / or a gyroscope), a microphone 730, an audio codec 735, a speaker 740, a communication circuit 745, a digital image capture circuit 750 (e.g., including a camera system), a video codec 755 (e.g., supporting a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. The multifunction electronic device 700 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0049] The processor 705 can execute instructions necessary for implementing or controlling the operation of many functions performed by the device 700 (e.g., such as the generation and / or processing of images as disclosed herein). The processor 705 can, for example, drive the display 710 and can receive user input from the user interface 715. The user interface 715 can allow the user to interact with the device 700. For example, the user interface 715 can take on various forms, such as buttons, keypads, dials, click-type touch wheels, keyboards, display screens, and / or touch screens. The processor 705 can also, for example, be a system on a chip, such as those present in mobile devices, and include a dedicated graphics processing unit (GPU). The processor 705 can be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture, and can include one or more processing cores. The graphics hardware 720 can be dedicated computing hardware for processing graphics and / or assisting the processor 705 in processing graphics information. In one embodiment, the graphics hardware 720 can include a programmable GPU.

[0050] Image capture circuitry 750 may include two (or more) lens assemblies 780A and 780B, each of which may have a separate focal length. For example, lens assembly 780A may have a shorter focal length relative to the focal length of lens assembly 780B. Each lens assembly may have a separate associated sensor element 790. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuitry 750 may capture still images and / or video images. Output from image capture circuitry 750 may be processed, at least in part, by video codec 755 and / or processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit or pipeline incorporated within image capture circuitry 750. The images thus captured may be stored in memory 760 and / or storage device 765.

[0051] The image capture circuitry 750 may capture still images and video images that may be processed, at least in part, in accordance with the present disclosure by the video codec 755 and / or the processor 705 and / or the graphics hardware 720, and / or a dedicated image processing unit incorporated within the image capture circuitry 750. The images thus captured may be stored in the memory 760 and / or the storage device 765. The memory 760 may include one or more different types of media used by the processor 705 and the graphics hardware 720 to perform device functions. For example, the memory 760 may include a memory cache, a read-only memory (ROM), and / or a random access memory (RAM). The storage device 765 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 765 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video disks (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The memory 760 and the storage device 765 may be used to tangibly hold computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 705, such computer program code may implement one or more of the methods described herein.

[0052] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, and tablet computers.

[0053] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the drawings in the present disclosure represent structures and devices in block diagram form to avoid obscuring the novel aspects of the disclosed concepts. For the sake of clarity, all features of the actual implementation may not be described. In addition, as part of this specification, some of the drawings in the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate one embodiment. In other embodiments, any of the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as part of the flow chart. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes and may not have been selected to delineate or limit the subject matter of the present invention, thereby resorting to the necessary claims to determine such subject matter. Reference throughout this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0054] It will be appreciated that in the development of any actual implementation (as in any software and / or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It will also be appreciated that such development work may be complex and time-consuming, but nonetheless, it is a routine undertaking for those of ordinary skill in the design and implementation of graphical modeling systems having the benefit of this disclosure.

[0055] This disclosure recognizes that the use of such personal information data within the present technology can be used to benefit users. For example, personal information data can be used to generate a virtual representation of the user in the form of an avatar. Furthermore, this disclosure contemplates other uses of personal information data that can benefit users. For example, health and fitness data can be used to provide insights into the user's overall health or as positive feedback to individuals using technology to pursue health goals.

[0056] This disclosure contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to robust privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be readily accessible to users and updated as data collection and / or use changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of those legitimate purposes. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent. Furthermore, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities with access to such personal information data comply with the other entities' privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific type of personal information data collected and / or accessed, as well as to applicable laws and standards, including jurisdictional considerations. For example, in the US, the collection of or access to certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices for different types of personal data should be maintained in each country.

[0057] It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide the material in the context of a specific embodiment, and its variation will be apparent to those skilled in the art (for example, some of the disclosed embodiments can be used in combination with each other). Therefore, the specific arrangement of the steps or actions shown or the arrangement of the elements should not be interpreted as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined with reference to the appended claims and the complete scope of the equivalents to such claims. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".

Claims

1. A method comprising: obtaining tracking data for a subject, the tracking data comprising a set of markers associated with an eye region; obtaining, for each eye of the subject, an iris region based on the set of landmarks associated with the eye region; determining a viewing direction of the subject; Get a lighting map for the environment; as well as Virtual representation data for a virtual representation of the subject is generated by applying a lighting effect to the iris region according to the lighting map and the viewing direction.

2. The method according to claim 1, wherein obtaining the iris region comprises: determining an eye region based on the set of markers, wherein the set of markers corresponds to a set of points on an eye opening, wherein the set of markers are each associated with position information; as well as The iris region within the eye region is identified based on color differences between pixels in image data including the eye region.

3. The method of claim 1, wherein the environment corresponds to an environment in which the virtual representation of the subject is to be presented. The method of claim 1 , wherein the environment corresponds to a physical environment in which the subject is located. The method of claim 1 , further applying lighting effects based on additional lighting maps for additional environments.

6. The method of claim 1, wherein the lighting effect comprises adjusting brightness to one or more regions of eyes of the virtual representation of the subject according to the lighting map.

7. The method of claim 1, wherein the lighting effect comprises reflecting a portion of the lighting map onto the iris region based on the viewing direction.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to: obtaining tracking data for a subject, the tracking data comprising a set of markers associated with an eye region; obtaining, for each eye of the subject, an iris region based on the set of landmarks associated with the eye region; determining a viewing direction of the subject; Get a lighting map for the environment; as well as Virtual representation data for a virtual representation of the subject is generated by applying a lighting effect to the iris region according to the lighting map and the viewing direction.

9. The non-transitory computer-readable medium of claim 8, wherein the computer-readable code for obtaining the iris region comprises computer-readable code for: determining an eye region based on the set of markers, wherein the set of markers corresponds to a set of points on an eye opening, wherein the set of markers are each associated with position information; and The iris region within the eye region is identified based on color differences between pixels in image data including the eye region.

10. The non-transitory computer-readable medium of claim 8, wherein the environment corresponds to an environment in which the virtual representation of the subject is to be presented.

11. The non-transitory computer-readable medium of claim 8, wherein the environment corresponds to a physical environment in which the subject is located.

12. The non-transitory computer readable medium of claim 8, further applying lighting effects based on additional lighting maps for additional environments.

13. The non-transitory computer-readable medium of claim 8, wherein the set of markers is obtained from sensor data captured by one or more sensors of a device worn by the subject, and Wherein the virtual representation data is generated based on the tracking data.

14. The non-transitory computer-readable medium of claim 8, wherein the lighting effect comprises adjusting brightness to one or more regions of eyes of the virtual representation of the subject according to the lighting map.

15. The non-transitory computer readable medium of claim 8, further comprising computer readable code for: An additional lighting effect is applied to a portion of the virtual representation of the subject that includes an eye region and excludes the iris region.

16. The non-transitory computer-readable medium of claim 8, wherein the viewing direction is determined based on a head pose.

17. The non-transitory computer-readable medium of claim 8, wherein the viewing direction is determined based on a gaze vector.

18. A system comprising: one or more processors; and One or more computer-readable media comprising computer-readable code executable by the one or more processors to: obtaining tracking data for a subject, the tracking data comprising a set of markers associated with an eye region; obtaining, for each eye of the subject, an iris region based on the set of landmarks associated with the eye region; determining a viewing direction of the subject; Get a lighting map for the environment; as well as Virtual representation data for a virtual representation of the subject is generated by applying a lighting effect to the iris region according to the lighting map and the viewing direction.

19. The system of claim 18, wherein the computer readable code for obtaining the iris region comprises computer readable code for: determining an eye region based on the set of markers, wherein the set of markers corresponds to a set of points on an eye opening, wherein the set of markers are each associated with position information; and The iris region within the eye region is identified based on color differences between pixels in image data including the eye region.

20. The system of claim 18, wherein the lighting effect comprises adjusting brightness to one or more regions of eyes of the virtual representation of the subject according to the lighting map.

Images