Hand cutout in response to low light conditions

Through dynamic management of hand cutout technology and hand tracking data to draw geometric shapes in low-light environments, the problem of hand detection under low-light is solved, and the accurate display and interaction quality of hands in mixed reality scenarios are achieved.

CN119068015APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410692921.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-28
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In low-light environments, hands are difficult to detect in mixed reality scenarios, resulting in the failure of extended reality technology functions, and the inability to accurately extract hand images, affecting the user's interaction with virtual content.

Method used

By dynamically managing hand cutout based on lighting conditions, using brightness thresholds to enable or disable hand cutout, combined with hand tracking data to draw geometric shapes under low light, providing visual indication of virtual content, ensuring that the hand is in the image Clear display.

Benefits of technology

Effectively identifying and displaying hand positions in low-light environments improves the quality of user interaction with virtual content, ensures the clarity and accuracy of hand images in mixed reality scenarios, and enhances the reliability of extended reality technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068015A_ABST
    Figure CN119068015A_ABST
Patent Text Reader

Abstract

The invention relates to hand matting in response to low light conditions. Managing hand cutout based on light conditions includes receiving image data including a hand, and determining brightness in the image data. If the brightness in the first image data satisfies a brightness threshold, a hand cutout is performed on the image data to extract a hand from the image data, and the image data is mixed with virtual content according to the extracted hand. If the brightness in the first image data satisfies a disabling threshold, the hand cutout is disabled, and the image data is mixed with the virtual content without extracting the hand. An alternative representation of the position of the hand is depicted in the final image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to image processing. More specifically, but not by way of limitation, the present disclosure also relates to techniques and systems for managing hand cutouts in response to lighting conditions in a scene. Background Art

[0002] With the rise of extended reality technologies, users are more frequently using see-through devices in which the view depicts virtual objects as well as physical objects. These devices typically allow the user's hands to be seen in the view of the mixed reality environment along with the virtual content. For example, some extended reality technologies allow users to use their hands to interact with virtual content.

[0003] A challenge is encountered when using XR in low-light environments and hands are visible in the scene. Features within the physical environment may be difficult to detect. This can make it difficult to extract the hand from the image, which may result in the functionality of the XR being modified. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1A to Figure 1B A flow diagram is shown of an example technique for managing hand matting in a mixed reality environment based on lighting conditions, according to one or more embodiments.

[0005] Figure 2 A flow diagram of a technique for generating a view of a hand in a mixed reality environment is shown according to one or more embodiments.

[0006] Figure 3 A state diagram illustrating a technique for managing hand cutouts based on lighting conditions in accordance with one or more embodiments.

[0007] Figure 4 An alternative state diagram of a technique for managing hand cutouts based on lighting conditions is shown according to some embodiments.

[0008] Figure 5 A flow diagram of a technique for blending an image of a hand with virtual content by drawing a geometric shape around the hand is shown in accordance with one or more embodiments.

[0009] Figure 6 An example diagram of a view with a composite image of camera content, virtual content, and geometry surrounding a hand is shown according to one or more embodiments.

[0010] FIG. 7A to FIG. 7B Depicted is an example flow diagram for blending image data captured by a camera with virtual content using a hand mesh in accordance with one or more embodiments.

[0011] Figure 8An example diagram of a view including a composite image of a rendered hand mesh and virtual content is shown according to one or more embodiments.

[0012] Fig. 9 An example diagram of a view including a composite image of a virtual representation of a hand and virtual content is shown according to one or more embodiments.

[0013] Fig.10 A simplified system diagram is shown in block diagram form according to one or more embodiments.

[0014] Fig.11 A computer system according to one or more embodiments is shown in block diagram form. DETAILED DESCRIPTION

[0015] The present disclosure generally relates to image processing. More specifically, but not by way of limitation, the present disclosure also relates to techniques and systems for managing hand cutouts in response to lighting conditions in a scene.

[0016] The present disclosure relates to systems, methods and computer-readable media for mixing image data with hands with virtual content image data. Specifically, the present disclosure relates to a technique for enabling or disabling hand cutouts based on the brightness level of a scene. According to some embodiments, an extended reality application can utilize hand cutouts to enhance the user's interaction with an extended reality environment. Hand cutout techniques can be used to identify hands from image data. The region of image data including the hand can be used to modify how the hand is presented in a mixed reality environment. For example, a user wearing a headset can interact with virtual content in the environment with their hands. Hand cutouts can be used to determine where the hand is located in a frame in order to modify the presentation of the physical environment and / or virtual content in order to provide user perspective information between the hand and the virtual content placed in the scene. According to some embodiments, hand cutouts involve applying image data to a model to obtain coefficients indicating the likelihood that a particular pixel or region of an image belongs to a hand. These coefficients can then be used to mix camera data from a current image and / or from an additional image frame with virtual content in a manner that provides a visual indication of a hand in space relative to virtual content in the scene.

[0017] Brightness can be determined in a variety of ways. For example, one or more lumen values ​​can be determined by image data of the physical environment, such as image data captured by a perspective camera. Brightness can be determined based on the brightness of a frame or a portion of a frame. For example, the brightness can be determined using the brightness of an area of ​​a frame including one or two hands. Alternatively or in addition, one or more lumen values ​​can be determined from a physical sensor that captures brightness values ​​by measuring scene brightness of the physical environment (such as readings obtained by an ambient light sensor). Alternatively or in addition, a separate model can measure scene brightness using either or both of image data and sensor readings. According to some embodiments, the operation of the hand cutout technique can be dynamically changed based on the characteristics of the scene. For example, if the total brightness of the frame drops to a low critical threshold, hand cutout can be disabled. However, if the total brightness of the frame drops below a different threshold, the brightness in the area of ​​the frame with a hand in the hand can be compared to one or more brightness thresholds. Based on the comparison, hand cutout will be utilized or disabled. According to some embodiments, when a certain threshold is met, computation can be saved by using only the region of the frame that includes hand data, thereby reducing the need to track the position of one or more hands in the frame. Alternatively, in some embodiments, another hand cutout model can be used for low light conditions. For example, another hand cutout model can be configured to perform hand cutout using images captured in low light or within a range of measured image brightness values.

[0018] Once hand cutout is disabled, hand cutout can be re-enabled based on one or more brightness metrics that may be the same or different from the brightness metrics used to disable hand cutout. For example, if the brightness of the frame is determined to meet the enablement threshold, hand cutout can be turned back on. In some embodiments, the brightness of the frame used to determine whether the enablement ritual is met can be the entire frame, or one or more regions of the frame including the hand. In some embodiments, when hand cutout is disabled to provide visual cues about the position of the hand relative to the virtual content in the scene, an alternative technique may be performed. For example, hand tracking data may provide position data of the hand in the image using different techniques and potentially different data than the hand cutout model. For example, hand tracking technology may provide three-dimensional position information of one or more points of the hand (such as joint positions). This position information can be used to draw geometric shapes around the hand in low light, and a low alpha value can be applied to the geometric shape so that the portion of the image data including the hand will be clearly visible in the resulting composite image.

[0019] Another alternative embodiment may include rendering the geometry of the hand in low light conditions. According to some embodiments, the hand tracking pipeline may generate a hand mesh. The mesh may be a three-dimensional geometric representation of the user's hand based on the joint positions. The mesh may be based on a general hand shape or a user-specific hand shape. In some embodiments, the mesh may be rendered so that the shape of the hand is presented in the composite image to provide visual cues about where the user's hand is located relative to the virtual content. In some embodiments, a user-specific mesh may be rendered that is specific to the geometry of the user's hand, and optionally, may be rendered using a user-specific texture so that the rendered mesh appears to be the user's hand.

[0020] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed concept. For the sake of clarity, all features of the actual specific implementation may not be described. In addition, as part of this specification, some of the drawings in the drawings of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate an embodiment. In other embodiments, any element in the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as part of the flow chart. In addition, the language used in the present disclosure has been selected primarily for readability and guidance purposes, and may not be selected to delimit or limit the subject matter of the present invention, or to resort to the necessary claims to determine such subject matter of the invention. References in this disclosure to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0021] It will be appreciated that in the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such a development effort may be complex and time-consuming, but will nevertheless be a routine undertaking for those of ordinary skill in the art who benefit from the image capture disclosed herein.

[0022] For purposes of this disclosure, the term "camera system" refers to one or more lens assemblies and one or more sensor elements and other circuitry for capturing images. For purposes of this disclosure, a "camera" may include more than one camera system, such as a stereo camera system, a multi-camera system, or a camera system capable of sensing the depth of a captured scene.

[0023] For purposes of this disclosure, the term "mixed reality environment" refers to a view of an environment that includes virtual content and physical content, such as image data from a perspective camera or some view of a physical environment.

[0024] Figure 1A A flow chart of a technique for managing hand cutouts in a mixed reality environment based on lighting conditions according to one or more embodiments is shown. For purposes of explanation, the processes described below will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0025] Flowchart 100 begins at block 105, where an image including a hand in a physical environment is received. The image data may be perspective camera data. For example, the electronic device may include a monocular camera, a stereo camera, and / or a depth sensor or other sensors that may be used to capture images of the physical environment and / or other sensor data for presentation on a display. In some embodiments, additional or alternative camera data may be used, such as data collected from an infrared camera or the like.

[0026] At block 107, flowchart 100 includes receiving virtual content to be mixed into the image data of the physical environment. According to one or more embodiments, the virtual content may be associated with one or more augmented reality applications and may be configured to be presented in a view of the physical environment. According to some embodiments, the virtual content may be generated locally on the device or may be received from a remote device.

[0027] Flowchart 100 continues at box 110, where image brightness is determined. As will be described below, image brightness can be determined in a variety of ways. For example, a brightness value can be determined from one or more pixels of a particular image frame. In some embodiments, a sample of pixels can be used for an image frame. For example, the total brightness of a pixel can be determined based on an overall, average or median brightness value of brightness, etc. In some embodiments, the brightness value can be used as a representative of brightness, or a predefined function can be used to convert it into a brightness value. In some embodiments, additional or alternative techniques can be used to determine brightness. For example, a camera that captures a view of a physical environment can determine various signals related to the brightness of a scene, such as automatic exposure. For another example, an ambient light sensor or other brightness sensor can provide a signal that can be used to determine the brightness of a scene.

[0028] Flowchart 100 continues to box 115, where it is determined whether the brightness value is less than a first brightness threshold. The first threshold may be a critical threshold brightness at which the total brightness of the scene is determined to be insufficient for hand cutout. For example, the first brightness threshold may indicate a brightness value at which it may be determined that the hand cutout technique may fail or be unreliable. If it is determined at box 115 that the brightness value is less than the first threshold, flowchart 100 ends at box 120 and hand cutout is disabled. According to some embodiments, by disabling hand cutout when the ambient lighting indicates that the obtained hand cutout data may fail or be unreliable, the amount of computation when generating a composite image in which the hand is presented is a. In some embodiments, the technique may be applied to any physical object in the image data. Instead of disabling hand cutout, extraction techniques for obtaining masks of physical objects may be enabled or disabled based on lighting.

[0029] Returning to box 115, if it is determined that the brightness value is not less than the first threshold, it is determined at box 125 whether the brightness is less than a second threshold. According to some embodiments, the second threshold may be a brightness value greater than the first threshold, but additional information is needed at this brightness value to determine whether the resulting hand cutout information will likely fail or be unreliable. In some embodiments, the overall brightness of the scene may not be so dark that the system can decisively determine that hand cutout should be disabled. Therefore, the second threshold is an intermediate threshold that indicates that additional data should be considered when the brightness value drops below the second threshold. Therefore, if it is determined that the area of ​​interest brightness is not less than the second threshold, the flowchart ends at box 140 and the image data is mixed with the virtual content using hand cutout. Conversely, if it is determined at box 125 that the brightness value is less than the second threshold, the flowchart proceeds to box 130.

[0030] At box 130, a hand bounding box is obtained that can determine the hand area of ​​the frame. In some embodiments, the system can run a hand tracking pipeline separate from the hand cutout pipeline. Image data that captures the surrounding environment can be used to extract hand features through the hand tracking pipeline. According to one or more embodiments, the network can be trained to read in sensor data (such as, a 2D image frame) and generate hand features (such as, a bounding box that depicts the hand area of ​​the image). The area of ​​the image data including the bounding box can be determined as a region of interest. In some embodiments, if the technology is performed for different physical objects, the bounding box can be determined to include the region of interest of the physical object.

[0031] Flowchart 100 proceeds to block 135 where the region of interest brightness value is compared to a third threshold. According to some embodiments, the region of interest brightness value may be determined based on the brightness values ​​of the pixels within the region of interest, as described above with respect to block 110. The pixels used to determine brightness at block 135 may include some of the pixels described above with respect to block 110, or may include additional pixels not previously considered at block 110.

[0032] If it is determined at box 135 that the area of ​​interest brightness is less than the third threshold, the flowchart ends at box 120 and hand cutout is disabled. To this end, the first threshold and the third threshold may be the same value, or may be different values. On the contrary, if it is determined that the area of ​​interest brightness is not less than the third threshold, the flowchart ends at box 140, and the image data is mixed with the virtual content using the hand cutout or other mask data associated with the physical object. According to some embodiments, hand cutout involves performing a hand cutout process to obtain or determine the attributes of the hand in the image data. According to some embodiments, the hand cutout process includes applying the image data to the model to obtain coefficients indicating the likelihood that a particular pixel or region of the image belongs to a hand. These coefficients can then be used to mix the image data from the current image frame and / or from the additional image frame with the virtual content obtained at box 107 in a manner that provides a visual indication of the hand in space relative to the virtual content in the scene. For example, coefficients determined for one or more specific frames of the image data stream can be used to mix the virtual content with another portion of the image data stream.

[0033] According to some embodiments, various thresholds (such as the first threshold in box 115, the second threshold in box 125, and the third threshold in box 135) can be based on static predefined values. Additionally or alternatively, one or more of the various thresholds may be dynamic values, or may be adaptive values. For example, one or more of the thresholds may be determined based on contextual characteristics of the environment, device, or user. These factors may include, for example, the user's enrollment data, hand cutout model accuracy, etc.

[0034] Figure 1B A flow chart of an alternative technique for managing hand cutouts in a mixed reality environment based on lighting conditions according to one or more embodiments is shown. For purposes of explanation, the processes described below will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0035] Similar to the above, flowchart 150 includes steps 105-115, as described above with respect to Figure 1A Briefly, flowchart 150 includes receiving image data including a hand in a physical environment at block 105. At block 107, receiving virtual content to be blended into a view of the physical environment. At block 110, determining an image brightness value.

[0036] Flowchart 100 continues to block 115, where it is determined whether the brightness value is less than a first brightness threshold. The first threshold may be a critical threshold brightness at which the overall brightness of the scene is determined to be insufficient to use hand cutouts. If it is determined at block 115 that the brightness is not less than the threshold, the flowchart proceeds to block 155.

[0037] At box 155, hand cutout is performed using the master model. According to some embodiments, hand cutout involves performing a hand cutout process to obtain or determine attributes of a hand in the image data. According to some embodiments, the hand cutout process includes applying the image data to the model to obtain coefficients indicating the likelihood that a particular pixel or area of ​​the image belongs to a hand. In some embodiments, multiple models may be available and configured to perform under different conditions. The master hand cutout model may be configured to work in a well-lit environment. These coefficients may then be used to mix image data from the current image frame and / or from additional image frames with the virtual content obtained at box 107 in a manner that provides a visual indication of the hand in space relative to the virtual content in the scene.

[0038] Returning to box 115, if it is determined that the brightness is not less than the threshold, the flowchart proceeds to box 160. At box 160, another hand cutout model can be used to generate coefficients. The alternative model can be configured to perform using low-light conditions, or other trade-offs about power and accuracy can be provided. For example, if the hand is present under conventional lighting, the hand cutout model working under low-light conditions may require more system resources to obtain a certain level of accuracy. As another example, the output of the alternative model may have a lower quality than the main hand cutout model. For example, the output of the alternative hand cutout model may be less accurate about the outline of the hand, resulting in a blurred hand template defined by the coefficient. These coefficients can then be used to mix image data from the current image frame and / or from the additional image frame with the virtual content obtained at box 107 in a manner that provides a visual indication of the hand in space relative to the virtual content in the scene.

[0039] Figure 2 A flow chart of a technique for generating a view of a hand in a mixed reality environment according to one or more embodiments is shown. For purposes of explanation, the processes described below will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0040] Pipeline 200 begins by collecting sensor data 202. Sensor data 202 can be collected by one or more cameras or other sensors of an electronic device (such as a head-mounted device or other wearable device or mobile device). For example, the electronic device may include a monocular camera, a stereo camera, and / or a depth sensor or other sensor that can be used to capture images and / or track objects in a scene. In some embodiments, the one or more cameras may include a scene camera configured to capture images of the environment in front of the device.

[0041] The pipeline includes a hand tracking module 204. The hand tracking module can use image data and / or depth data of the user's hand from the sensor data 202 to determine the characteristics of the hand. This can include, for example, joint positions, postures, depths, positions, orientations, etc. For example, a visual-based tracking process is used to monitor one or more hands of the user. The visual-based tracking process can utilize image data captured by a camera of an electronic device. The hand tracking module 204 may include a hand tracking network trained to estimate the physical state of one or more hands of the user. In some embodiments, the hand tracking module 204 may provide a bounding box 208 for each hand or both hands in the view. The bounding box 208 may be determined for each frame of the sensor data 202, or the bounding box may be determined when triggered based on the lighting characteristics of the environment. In some embodiments, the hand tracking module 204 may be configured to generate a representation of the geometry of the hand gesture, for example in the form of a grid. The grid may be a general hand grid representing the gesture of the hand, or may be user-specific. For example, in some embodiments, the hand tracking module 204 can be configured to use user-specific geometry (eg, from enrollment data) to represent hand gestures.

[0042] In some embodiments, a hand mask 210 may be determined using a hand cutout module 206. The hand mask 210 may be the shape or geometry of a hand and / or arm recognized by a user device. The hand mask 210 may be determined based on a model trained to predict a hand mask from the sensor data 202.

[0043] The synthesizer 220 also receives image data from which a synthesized image is generated. Specifically, the sensor data 202 may include perspective camera data 212 that provides a view of the environment. The perspective camera data 212 may be associated with a brightness value 214 derived from the brightness of the pixels included in the perspective camera data. In addition, virtual image data 218 to be presented in the view of the environment may be provided. The virtual image data 218 may be generated from an application on a local device, may be received from a mobile device, or some combination thereof. The synthesizer 220 may be configured to blend the virtual image data 218 into the perspective camera data 212 to generate a synthesized frame 224. If the hand cutout module 206 is activated and a hand mask 210 is provided, the synthesizer may blend the virtual image data 218 with the perspective camera data 212 according to the hand animation cutout module 206.

[0044] Various techniques can be used to enable and disable hand cutouts based on environment lighting. Figure 3 A state diagram of a technique for managing hand cutouts based on lighting conditions according to one or more embodiments is shown. Specifically, Figure 3The state machine of FIG. 1 depicts a technique for initiating and terminating a low-light mitigation feature that is configured to trigger enabling and disabling of hand cutouts.

[0045] State diagram 300 begins at box 302, where low light mitigation is turned off. When similar mitigation is turned off, hand matting is enabled. Thus, a synthetic image that blends virtual content into a view of a physical environment can be generated based on a hand mask provided by the hand matting pipeline. The hand matting pipeline can be configured to determine a hand mask based on image data. The image data can be perspective image data, and / or can be image data captured by an alternative camera source such as an infrared camera or other camera. In some embodiments, if the hand mask is determined using an alternative camera source, additional image data can be obtained from the alternative camera source simultaneously with the perspective camera. The perspective camera data can be reprojected to the additional camera data, and the hand can be extracted from the perspective camera data based on the reprojection.

[0046] When hand cutout is turned on and low light mitigation is turned off, a scene brightness value can be determined. As described above, the scene brightness value can be determined in a variety of ways. For example, the brightness value can be derived from the brightness of a pixel or sampled pixel in a frame of sensor data that captures the scene. If the scene brightness is less than the critical enable threshold, the state diagram transitions to box 308, where the system is determined to be in a critical low light state. This may occur, for example, when the brightness level of the scene is deemed so dark that a hand mask generated by the hand cutout pipeline using the image data of the scene may be unreliable. During the critical low light state at box 308, hand cutout is turned off or otherwise disabled in accordance with some embodiments.

[0047] When the system is in a critical low light state, as shown at block 308, it may be determined whether the scene brightness for a given frame is greater than a critical disable threshold. According to some embodiments, the critical disable threshold may indicate a threshold brightness value at which the brightness level of the scene is deemed bright enough so that a hand mask generated by the hand matting pipeline using image data of the scene may be reliable. When it is determined that the scene brightness exceeds the critical disable threshold, then the state diagram 300 may transition back to block 302, where low light mitigation is turned off and hand matting is enabled. In some embodiments, the critical enable threshold and the critical disable threshold may be different values, or may be the same brightness value.

[0048] From box 302, the scene brightness can be compared to a scene brightness enable threshold that is different from a critical threshold. The scene brightness enable threshold can be a brightness value greater than the critical threshold, but additional information is needed at this threshold to determine whether the resulting hand cutout information will likely fail or be unreliable. Therefore, if it is determined that the scene brightness drops below the scene brightness enable threshold, the state diagram 300 transitions to box 304 and the hand bounding box is tracked. In some embodiments, from box 302, if it is determined that the scene brightness is below the critical threshold, the scene brightness may also be below the scene brightness enable threshold. However, the determination of whether the scene brightness level is the critical disable threshold will override the determination that the scene brightness drops below the scene brightness enable threshold.

[0049] At box 304, a hand bounding box is tracked. This may include, for example, enabling or starting a hand tracking system to begin determining a bounding box around one or more of the hand scene image data in the hand scene image data. The bounding box may correspond to a hand region of interest in the image data. According to one or more embodiments, a brightness value within the hand region of interest is determined, and hand cutout is turned on. Thus, at box 304, the system may simultaneously determine a brightness value for the region of interest and a scene brightness value. In some embodiments, the brightness of the region of interest and the scene brightness may be based on a set of identical brightness values ​​for sampled pixels within the image data. However, in some embodiments, different pixels may be used for each determination. Thus, if at box 304, the scene brightness exceeds a scene brightness disable threshold, the state diagram 300 may transition back to box 302, and low light mitigation may be turned off. In addition, hand cutout will be enabled. Thus, a composite image that mixes virtual content with perspective camera data may be generated based on a hand mask generated by hand cutout.

[0050] If it is determined at box 304 that the region of interest brightness drops below the region of interest enable threshold, the state diagram 300 transitions to box 306 and low light mitigation is turned on. Activating low light mitigation includes deactivating hand matting. Thus, for image frames where the region of interest brightness drops below the region of interest enable threshold, a synthetic image can be generated by blending with perspective scene data in virtual content without considering the hand mask. In some embodiments, a virtual hand can be rendered using hand data (such as hand joints, hand positions, etc.) available from a hand tracking network. In some embodiments, a virtual hand can be rendered as an alternative to using a bounding box. When low light mitigation is turned on and hand matting is turned off, the brightness value of the region of interest can continue to be tracked across additional frames. When it is determined that the region of interest brightness exceeds the region of interest disable threshold, the state diagram 300 returns to box 304 and the hand matting is turned on. In addition, the hand region of interest brightness and the scene brightness continue to be tracked, and it is determined whether the state diagram transitions to disable low light mitigation at box 302, and thus stop tracking the bounding box. Alternatively, from block 304, another determination is made that the region of interest brightness drops below the region of interest enable threshold, and the state diagram 300 returns to block 306, where low light mitigation is turned on and hand cutout is turned off.

[0051] Figure 4 An alternative state diagram of a technique for managing hand cutouts based on lighting conditions according to some embodiments is shown. Specifically, Figure 3 The state machine depicts the state machine for starting and terminating a Figure 3 The described thresholds use different thresholds to trigger a technique for enabling and disabling the low-light mitigation feature of the hand cutout.

[0052] Similar to the above Figure 3 , state diagram 400 begins at block 302, where low light mitigation is off. When similar mitigation is off, hand matting is enabled. Thus, a composite image that blends virtual content into a view of a physical environment may be generated based on a hand mask provided by the hand matting pipeline. From block 302, scene brightness may be compared to a scene brightness enable threshold. If it is determined that scene brightness has dropped below the scene brightness enable threshold, state diagram 400 transitions to block 304.

[0053] At box 304, the hand bounding box is tracked. This may include, for example, enabling or starting the hand tracking system to begin determining a bounding box around one or more of the hand scene image data. If at box 304, the scene brightness exceeds the scene brightness disable threshold, the state diagram 400 may transition back to box 302 and low light mitigation may be turned off. In addition, hand matting will be enabled. Thus, a composite image that blends virtual content with perspective camera data may be generated based on a hand mask generated by hand matting. If it is determined at box 304 that the region of interest brightness drops below the region of interest enable threshold, the state diagram 400 transitions to box 306 and low light mitigation is turned on. Activating low light mitigation includes deactivating hand matting. When it is determined that the region of interest brightness exceeds the region of interest disable threshold, the state diagram 400 returns to box 304 and the hand matting is turned on. In addition, the hand region of interest brightness and the scene brightness continue to be tracked, and it is determined whether the state diagram transitions to disable low light mitigation at box 302, and thus stop tracking the bounding box.

[0054] Returning to block 302, if the scene brightness is less than the critical enable threshold while low light mitigation is off, the state diagram transitions to block 408, where the system is determined to be in a critical low light state. When the system is in the critical low light state, a hand bounding box is tracked while hand matting is off. This may include, for example, enabling or starting a hand tracking system to begin determining a bounding box around one or more of the hand scene image data. However, the hand matting pipeline will not generate a hand mask for these frames. The brightness of the region of interest based on the bounding box may be determined.

[0055] When the system is in a critical low light state, it may be determined whether the scene brightness of a given frame is greater than a critical enable threshold. When it is determined that the scene brightness exceeds the critical disable threshold, then state diagram 400 may transition back to block 302 and low light mitigation may be turned off. Conversely, if it is determined at block 408 that the region of interest brightness is below the region of interest enable threshold, then state diagram 400 transitions to block 306 and low light mitigation is turned on.

[0056] In some embodiments, an alternative technique may be performed when hand cutouts are disabled to provide visual cues about the position of the hands relative to virtual content in the scene. Figure 5 , depicts a flow chart of a technique for blending an image of a hand with virtual content by drawing a geometric shape around the hand. For purposes of explanation, the processes described below will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in a different order. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0057] Flowchart 500 begins with steps 105-115, as described above with respect to FIG1. ​​Briefly, flowchart 500 includes receiving image data including a hand in a physical environment at box 105. At box 107, receiving virtual content to be blended into a view of the physical environment. At box 110, determining an image brightness value.

[0058] Flowchart 100 continues to block 115, where it is determined whether the brightness value is less than a first threshold. The first threshold may be a critical threshold brightness at which the overall brightness of the scene is determined to be insufficient to use hand cutout. If it is determined at block 115 that the brightness is not less than the threshold, the flowchart proceeds to block 525. At block 525, hand cutout is performed. According to some embodiments, hand cutout involves performing a hand cutout process to obtain or determine attributes of a hand in the image data. According to some embodiments, the hand cutout process includes applying the image data to a model to obtain coefficients indicating the likelihood that a particular pixel or region of the image belongs to a hand.

[0059] The flowchart then ends at block 530, where the image data is blended with the virtual content based on the hand matte. According to one or more embodiments, the coefficients are used to blend the image data from the current image frame and / or from the additional image frame with the virtual content obtained at block 107 in a manner that provides a visual indication of the hand in space relative to the virtual content in the scene.

[0060] Returning to block 115, if it is determined that the brightness value is less than the threshold, flowchart 500 proceeds to block 535 and the hand position is determined. According to one or more embodiments, the hand position can be determined as a point or area in space representing the position of the hand. In some embodiments, the hand position can be determined based on hand tracking data, which can use additional or alternative sensor data to track the user's hand in addition to the hand cutout process. As described above with respect to Figure 2 As described, the hand tracking module may use image data and / or depth data of the user's hand from the sensor data 202 to determine characteristics or attributes of the hand. This may include, for example, joint positions, posture, depth, position, orientation, etc. The hand tracking network used to obtain hand tracking data may be different from the model used for hand matting and may be configured to obtain different inputs.

[0061] Flowchart 500 proceeds to block 540, where the device draws a geometric shape based on the hand position. According to one or more embodiments, the geometric shape can be a plane in space configured to follow the determined position of the hand. In some embodiments, the geometric shape can be configured to surround all or part of the hand, fingertips, or other parts of the user.

[0062] Flowchart 500 continues at block 545, where one or more low alpha values ​​are assigned to the geometric shape. According to one or more embodiments, the alpha value is assigned to the content to be blended into the composite frame and indicates the transparency level of the corresponding portion of the frame. For example, the alpha value may be applied to a specific pixel or region of the content. By rendering the geometric shape surrounding the hand with a low alpha value, multiple perspective camera image data in the perspective camera image data under the geometric shape becomes visible. For example, the low alpha value may be a predefined alpha value, or may be an alpha value assigned to be below a predefined threshold.

[0063] The flowchart ends at block 550, where the image data from the perspective camera is blended with the virtual content based on the drawn geometric shape. That is, the area of ​​the frame that includes the geometric shape will show multiple camera image data in the camera image data of the physical environment, thereby providing a visual cue of the position of the hand relative to the virtual content in the augmented reality scene.

[0064] Figure 6 An example diagram of a view of a composite image with camera content and virtual content according to one or more embodiments is shown. This example diagram is intended to provide information about the above Figure 5 The process described is for clarity of an example result and is not intended to limit the above Figure 5 Describe the process.

[0065] exist Figure 6 In the diagram of FIG, a display 600 depicting an extended reality image is shown, which includes camera image data 605 of a physical environment and virtual content 610 in the form of user interface components. As shown, the physical environment is depicted in low light, which may make hand cutout difficult or ineffective. Thus, hand tracking can be used to identify the position of the hand 615. A geometric shape 620 is then drawn around the determined position of the hand 615 with a low alpha value so that the hand 615 is visible on the virtual content 610. In some embodiments, the geometric shape 620 can be drawn at a depth corresponding to the depth of the hand 615, so that the relative position of the geometric shape 620 and the virtual content 610 can be clearly seen from the way the virtual content 610 and the geometric shape 620 are rendered (for example, if the geometric shape is in front of or behind the virtual content).

[0066] In some embodiments, a virtual hand may be rendered to provide a visual indication of the position of the user's hand relative to virtual content in the scene. FIG. 7A to FIG. 7BDepicted is an example flow chart for blending image data captured by a camera with virtual content using a hand mesh according to one or more embodiments. For purposes of explanation, the processes described below will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in a different order. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0067] Flowchart 700 begins with steps 105-115, as described above with respect to Figure 1A Briefly, flowchart 500 includes receiving image data including a hand in a physical environment at block 105. At block 107, receiving virtual content to be blended into a view of the physical environment. At block 110, determining an image brightness value.

[0068] Flowchart 100 continues to block 115, where it is determined whether the brightness value is less than a first threshold. The first threshold may be a critical threshold brightness at which the overall brightness of the scene is determined to be insufficient to use hand cutout. If it is determined at block 115 that the brightness is not less than the threshold, the flowchart proceeds to block 735. According to some embodiments, hand cutout involves performing a hand cutout process to obtain or determine attributes of a hand in the image data. According to some embodiments, the hand cutout process includes applying the image data to a model to obtain coefficients indicating the likelihood that a particular pixel or region of the image belongs to a hand.

[0069] The flowchart then ends at block 740, where the image data is blended with the virtual content based on the hand matte. According to one or more embodiments, the coefficients are used to blend the image data from the current image frame and / or from the additional image frame with the virtual content obtained at block 107 in a manner that provides a visual indication of the hand in space relative to the virtual content in the scene.

[0070] Returning to block 115, if it is determined that the brightness value is less than the threshold, flowchart 700 proceeds to block 720 and the hand position is determined. According to one or more embodiments, the hand position can be determined as a point or area in space representing the position of the hand. In some embodiments, the hand position can be determined based on hand tracking data, which can use additional or alternative sensor data to track the user's hand in addition to the hand cutout process. As described above with respect to Figure 2 As described, the hand tracking module may use image data and / or depth data of the user's hand from the sensor data 202 to determine characteristics or attributes of the hand. This may include, for example, joint positions, posture, depth, position, orientation, etc. The hand tracking network used to obtain hand tracking data may be different from the model used for hand matting and may be configured to obtain different inputs.

[0071] Flowchart 700 proceeds to block 725 where a hand mesh is obtained. The hand mesh may be obtained as a Figure 2 The hand tracking module 204 may be configured to generate a geometric representation of the hand gesture, such as in the form of a mesh, a point cloud, a geometric volume, etc. The geometric representation may be a general hand geometry representing the hand gesture, or may be user specific.

[0072] The flowchart proceeds to block 730 where the image data from the perspective camera is blended with the virtual content based on the mesh and the hand tracking data. That is, a virtual representation of the hand is rendered and presented in the perspective image data at a location corresponding to the real user's hand to provide a visual indication of the placement of the user's hand in the scene relative to the other virtual content.

[0073] Go to Figure 7B , describes the process of blending image data from block 730 in more detail. The flowchart begins at block 755, where a hand mesh is rendered based on hand tracking data. As described above, the hand mesh may be provided by the hand tracking module 204. Additionally or alternatively, the system may be configured to derive a hand mesh representing the geometry of the hand's pose based on the hand tracking data from the hand tracking module 204.

[0074] Optionally, as shown at block 760, a user-specific hand mesh and / or a user-specific hand texture may be obtained. According to one or more embodiments, a user-specific hand mesh and / or a user-specific hand texture may be determined based in part on the enrollment data. For example, the enrollment process may include a prompt to enroll the user's hands so that the system can capture a representative view of one or more of the user's hands from which a user-specific geometry and texture may be determined. The texture and geometry may then be retried and used to determine the current user-specific mesh and texture. The flowchart may then optionally proceed to block 765, where a virtual representation of the hand is rendered using the user-specific hand mesh and hand texture. According to the hand tracking data.

[0075] The flowchart proceeds to block 770 where the hand mesh is reprojected from the hand tracking frame rate to the display rate. According to one or more embodiments, the hand tracking data may be captured at a frame rate different from the frame rate at which the composite image is to be displayed. Thus, reprojection may be performed to generate a rendered hand mesh for each frame to be displayed.

[0076] The flowchart ends at block 775, where the reprojected hand mesh is then blended with other virtual content. Figure 2The compositor 220 can manage virtual image data 218 to be rendered in a view of the physical environment and a rendered mesh to form a composite image in which a virtual representation of a hand in the form of a rendered hand mesh and / or hand texture is presented in the view based on the hand tracking data so as to provide visual cues as to the relationship between the user's hand and the virtual content from box 107.

[0077] Figure 8 An example diagram of a view of a composite image including a rendered hand mesh and virtual content according to one or more embodiments is shown. This example diagram is intended to provide information about the above FIG. 7A to FIG. 7B The process described is for clarity of an example result and is not intended to limit the above FIG. 7A to FIG. 7B Describe the process.

[0078] exist Figure 8 In the diagram of FIG. 6 , a display 600 depicting an extended reality image is shown, which includes camera image data 605 of a physical environment and virtual content 610 in the form of a user interface component. As shown, the physical environment is depicted in low light, which may make hand cutout difficult or ineffective. Thus, hand tracking can be used to identify the position of the hand. For example, a hand tracking module can generate a hand mesh. The mesh can be a three-dimensional geometric representation of the user's hand based on the joint position based on a general hand geometry or a user-specific hand geometry. In the current example, a rendered mesh 820 representing the placement of the hand based on the posture and position of the hand determined from the hand tracking data is presented. Because the rendered mesh 820 is a virtual object, it can be mixed with other virtual content 610 according to the position information of each component in the component. For example, if the hand is closer to the user than the virtual content 610, the rendered mesh 820 will appear above the virtual content. On the contrary, if the hand is behind the virtual content, the rendered mesh 820 can appear behind the virtual content 610.

[0079] Fig. 9 An alternative example diagram is depicted in which a virtual hand 920 is rendered based on user-specific hand attributes. Fig. 9, a display 600 depicting an extended reality image is shown, which includes camera image data 605 of the physical environment and virtual content 610 in the form of user interface components. Similarly, hand tracking can be used to identify the position of the hand. For example, the hand tracking module can generate a hand mesh. The mesh can be a three-dimensional geometric representation of the user's hand based on the joint position based on the user-specific hand geometry. For example, the system can be configured to track the joint position or other attributes of the hand, and generate a user-specific mesh using predefined user-specific data (such as registration data) for the user-specific hand geometry. In addition, the system can be configured to use predefined user-specific data for user-specific hand textures to render textures on the user-specific mesh. Therefore, the virtual hand 920 is presented in a more visible way to the user.

[0080] In addition, since the virtual hand 920 is a virtual object, it can be mixed with other virtual content 610 according to the position information of each of the parts. For example, if the hand is closer to the user than the virtual content 610, the virtual hand 920 will appear above the virtual content. On the contrary, if the hand is behind the virtual content, the virtual hand 920 may appear behind the virtual content 610.

[0081] refer to Fig.10 , which is a simplified block diagram of an electronic device 1000 that can be used to generate and display a mixed reality scene. The system diagram includes an electronic device 1000, which can include various components. The electronic device 1000 can be part of a multi-function device such as a phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted device, a base station, a laptop computer, a desktop computer, a network device, or any other electronic device capable of capturing image data.

[0082] The electronic device 1000 may include one or more processors 1030, such as a central processing unit (CPU). The processor 1030 may include a system on a chip (such as those present in mobile devices), and may include one or more dedicated graphics processing units (GPUs) or other graphics hardware. In addition, the processor 1030 may include multiple processors of the same or different types. The electronic device 1000 may also include a memory 1040. The memory 1040 may include one or more different types of memory, which may be used to perform device functions in conjunction with the processor 1030. The memory 1040 may store various programming modules for execution by the processor 1030, including a tracking module 1085 and various other applications 1075 that may generate virtual content. The electronic device 1000 may also include a storage device 1050. The storage device 1050 may include data utilized by the tracking module 1085 and / or the application 1075. For example, the storage device 1050 may be configured to store user profile data, media content to be displayed as virtual content, etc., such as displayed on the display 1080.

[0083] In some embodiments, the electronic device 1000 may include other components for vision-based tracking, such as one or more cameras 1010 and / or other sensors, such as one or more depth sensors 1060. In one or more embodiments, each of the one or more cameras 1010 may be a traditional RGB camera, an IR camera, a depth camera, etc. In addition, the camera 1010 may include a stereo or other multi-camera system, a time-of-flight camera system, etc., which capture images from which the depth information of the scene can be determined. The camera 1010 may include a camera that is incorporated into the electronic device 1000, for example, to capture different areas from different perspectives. For example, the camera 1010 may include one or more scene cameras and one or more user-facing cameras, such as an eye tracking camera or a body tracking camera. The electronic device 1000 may additionally include one or more I / O devices 1020, which may be configured to allow a user to interact with the electronic device, such as a keyboard, an input component, a speaker, etc.

[0084] In one or more embodiments, the tracking module 1085 can track user characteristics, such as the position of the hands. The tracking module 1085 can determine whether the user's hands are occupied, such as when the user is interacting with physical objects in the environment.

[0085] Although electronic device 1000 is depicted as including numerous components as described above, in one or more embodiments, the various components and functionality of the components may be distributed in different ways across one or more additional devices (e.g., across a network). In some embodiments, any combination of data or applications may be deployed partially or completely on additional devices such as network devices, network storage devices, etc. Similarly, in some embodiments, the functionality of tracking module 1085 and application 1075 may be deployed partially or completely on additional devices across a network.

[0086] In addition, in one or more embodiments, the electronic device 1000 can be composed of multiple devices in the form of an electronic system. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted. In one or more embodiments, various calls and transmissions can be directed differently based on the functions of different distributions. In addition, additional components can be used, and certain combinations of the functions of any component can also be combined.

[0087] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0088] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0089] Reference now Fig.11 , which shows a simplified functional block diagram of an exemplary multifunctional electronic device 1100 according to one embodiment. Each electronic device in the electronic device may be a multifunctional electronic device, or may have some or all of the components described in the multifunctional electronic device described herein. The multifunctional electronic device 1100 may include a processor 1105, a display 1110, a user interface 1115, graphics hardware 1120, device sensors 1125 (e.g., proximity sensors / ambient light sensors, accelerometers and / or gyroscopes), microphones 1130, audio codecs 1135, one or more speakers 1140, communication circuits 1145, digital image capture circuits 1150 (e.g., including a camera system), memory 1160, storage devices 1165, and some combination of communication buses 1170. The multifunctional electronic device 1100 may be, for example, a mobile phone, a personal music player, a wearable device, a tablet computer, etc.

[0090] Processor 1105 can execute necessary instructions to implement or control the operation of multiple functions performed by device 1100. Processor 1105 can, for example, drive display 1110 and can receive user input from user interface 1115. User interface 1115 can allow user to interact with device 1100. For example, user interface 1115 can take various forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, touch screens, etc. Processor 1105 can also be a system on chip, such as those present in mobile devices, and include a dedicated graphics processing unit (GPU). Processor 1105 can be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture, and can include one or more processing cores. Graphics hardware 1120 can be dedicated computing hardware for processing graphics and / or assisting processor 1105 to process graphics information. In one embodiment, graphics hardware 1120 can include a programmable GPU.

[0091] The image capture circuit 1150 may include one or more lens assemblies, such as 1180A and 1180B. The lens assemblies may have a combination of various characteristics, such as different focal lengths, etc. For example, relative to the focal length of lens assembly 1180B, lens assembly 1180A may have a short focal length. Each lens assembly may have a separate associated sensor element 1190. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 1150 may capture still images, video images, enhanced images, etc. The output from the image capture circuit 1150 may be processed at least in part by: a video codec 1155 and / or a processor 1105 and / or graphics hardware 1120 and / or a dedicated image processing unit or pipeline incorporated into the circuit 1145. The images captured in this way may be stored in the memory 1160 and / or the storage device 1165.

[0092] Memory 1160 may include one or more different types of media used by processor 1105 and graphics hardware 1120 to perform device functions. For example, memory 1160 may include memory cache, read-only memory (ROM) and / or random access memory (RAM). Storage device 1165 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage device 1165 may include one or more non-transitory computer-readable storage media, which include, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). Memory 1160 and storage device 1165 may be used to tangibly retain computer program instructions or computer-readable codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 1105 , such computer program code may implement one or more of the methods described herein.

[0093] It should be understood that the above description is intended to be illustrative and not limiting. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in conjunction with each other). Thus, FIG. 1 to FIG. Figure 5 and FIG. 7A to FIG. 7B The specific arrangement of steps or actions shown in FIG. Figure 6 , Figure 8 and Figure 10 to Figure 11 The arrangement of elements shown in the accompanying drawings should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".

Claims

1. A computer program product comprising computer readable code executable by one or more processors to: receiving a stream of image data including a hand; According to a first brightness value of a first portion of the image data stream satisfying a first brightness threshold, performing a hand cutout process, wherein the image data stream is used to identify attributes of the hand; and The image data stream is mixed with virtual content according to the attributes of the hand.

2. The computer program product of claim 1 , further comprising computer readable code for: Based on a determination that a second brightness value for a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is disabled.

3. The computer program product of claim 1 , wherein the hand cutout process is performed using a first hand cutout model based on the first brightness value satisfying the first brightness threshold, and the computer program product further comprises computer readable code for performing the following operations: Based on a determination that a second brightness value of a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is performed on the image data stream using a second hand matting model.

4. The computer program product of claim 1 , further comprising computer readable code for: Based on the determination that a second brightness value of the second portion of the image data stream fails to satisfy a second brightness threshold: determining a hand region in said second portion of said image data stream, determining the brightness of the hand region, and Based on a determination that the brightness in the hand region of the second portion of the image data stream fails to satisfy a third brightness threshold, the hand matting process is disabled.

5. The computer program product of claim 4, further comprising computer readable code for: Based on the determination that the third brightness value of the third portion of the image data stream fails to satisfy the fourth brightness threshold: Enables the hand matting process. The computer program product of claim 5 , wherein the fourth brightness threshold is greater than the third brightness threshold.

7. The computer program product of claim 1, wherein the first brightness value of the first portion of the image data stream is determined based on a median brightness of a set of pixels of the first portion of the image data stream.

8. The computer program product of claim 1, wherein the computer readable code for executing the hand cutout process comprises computer readable code for: selecting an alternative camera source based on the first brightness value of the first portion of the image data stream; obtaining additional image data from the alternative camera source; reprojecting the first portion of the image data stream to the additional image data; and Attributes of the hand are identified from the first portion of the image data stream based on the re-projection.

9. A system, comprising: one or more processors; and One or more computer readable media including computer readable code executable by the one or more processors to: receiving a stream of image data including a hand; According to a first brightness value of a first part of the image data stream satisfying a first brightness threshold, performing a hand cutout process, wherein the image data stream is used to identify attributes of the hand in the process; as well as The image data stream is mixed with virtual content according to the attributes of the hand.

10. The system of claim 9, further comprising computer readable code for: Based on a determination that a second brightness value for a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is disabled.

11. The system of claim 9, wherein the hand matting process is performed using a first hand matting model based on the first brightness value satisfying the first brightness threshold, and the system further comprises computer readable code for performing the following operations: Based on a determination that a second brightness value of a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is performed on the image data stream using a second hand matting model.

12. The system of claim 9, further comprising computer readable code for: Based on the determination that a second brightness value of the second portion of the image data stream fails to satisfy a second brightness threshold: determining a hand region in said second portion of said image data stream, determining the brightness of the hand region, and Based on a determination that the brightness in the hand region of the second portion of the image data stream fails to satisfy a third brightness threshold, the hand matting process is disabled.

13. A method comprising: receiving a stream of image data including a hand; According to a first brightness value of a first part of the image data stream satisfying a first brightness threshold, performing a hand cutout process, wherein the image data stream is used to identify attributes of the hand in the process; as well as The image data stream is mixed with virtual content according to the attributes of the hand.

14. The method according to claim 13, further comprising: Based on a determination that a second brightness value for a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is disabled.

15. The method according to claim 13, wherein according to the first brightness value satisfying the first brightness threshold, the hand cutout process is performed using a first hand cutout model, and the method further comprises: Based on a determination that a second brightness value of a second portion of the image data stream fails to satisfy a first brightness threshold, the hand matting process is performed on the image data stream using a second hand matting model.

16. The method according to claim 13, further comprising: Based on the determination that a second brightness value of the second portion of the image data stream fails to satisfy a second brightness threshold: determining a hand region in said second portion of said image data stream, determining the brightness of the hand region, and Based on a determination that the brightness in the hand region of the second portion of the image data stream fails to satisfy a third brightness threshold, the hand matting process is disabled.

17. The method according to claim 16, further comprising: Based on the determination that the third brightness value of the third portion of the image data stream fails to satisfy the fourth brightness threshold: Enables the hand matting process. The method of claim 17 , wherein the fourth brightness threshold is greater than the third brightness threshold.

19. The method of claim 13, wherein the first luminance value of the first portion of the image data stream is determined based on a median brightness of a set of pixels of the first portion of the image data stream.

20. The method of claim 13, wherein performing the hand cutout process comprises: selecting an alternative camera source based on the first brightness value of the first portion of the image data stream; obtaining additional image data from the alternative camera source; reprojecting the first portion of the image data stream to the additional image data; as well as Attributes of the hand are identified from the first portion of the image data stream based on the re-projection.