Fixed photonic-to-photonic programmable display pipeline

Abstract

Various implementations disclosed herein include devices, systems, and methods to implement pass-through effects via use of programmable elements (e.g., GPUs) in a captured image-rendered content combination pipeline. This can involve implementing a programmable block in a captured image-rendered content combination pipeline that can apply effects to pass-through video between an ISP (e.g., a pass-through source) and fixed display pipeline hardware, in which captured image data is processed (e.g., modified and combined with rendered content) in stages that must begin / end according to timing constraints in order to ensure timely display (i.e., live pass-through via beam tracking) (i.e., less than a full frame is processed at a time).

Description

Technical Field

[0001] This disclosure relates in its entirety to systems, methods, and apparatuses for providing extended reality (XR) views, thereby blending virtual content with transparent video content on electronic devices such as head-mounted displays (HMDs). Background Technology

[0002] Passthrough video provides a live view based on images captured by one or more cameras in a device's camera system. Devices such as HMDs can provide passthrough video that allows users to "see through" the device's display to view the view based on the physical surroundings. This passthrough video may involve cameras visually capturing images of the physical environment to provide a "live feed" using one or more processes to correct the viewing angle and reduce distortion. Passthrough devices can also provide XR experiences by combining (e.g., blending) rendered virtual content with passthrough video. Existing passthrough devices may be limited in terms of the programmable effects that can be achieved within their display pipeline. Summary of the Invention

[0003] The various specific implementations disclosed herein include apparatuses, systems, and methods for achieving pass-through effects via the use of programmable elements (e.g., GPUs) in a pipeline that combines captured images with rendered virtual content. More specifically, in a captured image-rendered content combining pipeline, a programmable block is implemented between the ISP (pass-through source) and fixed display pipeline hardware, in which captured image data is processed (e.g., modified and combined with rendered content) in stages that must begin / end according to timing constraints to ensure timely display (i.e., live pass-through via beam tracking) (i.e., less than a full frame processed at a time). This programmable block is configured to apply effects to the pass-through video.

[0004] In some implementations, the electronic device performs a method (e.g., via one or more processors). This method performs one or more steps or processes. In some implementations, the method involves acquiring image data comprising frames of images sequentially captured in a physical environment by an image capture device on the electronic device. The method involves generating blended frames by sequentially processing segments of frames via a programmable element (e.g., a GPU or other element in which a non-hardware-fixed instruction set can be executed). Image frames may be broken down into such smaller segments to facilitate speed (e.g., reduce the time from image capture to display), and thus each segment processed by the programmable element may comprise fewer than a full frame. The processing includes performing one or more visual effects on the segments (e.g., by executing instructions not fixed in hardware) and blending the segments with rendered content. One or more visual effects may be used to modify the segments (e.g., coloring, adding modifications (e.g., adding notes as part of a wall), removing modifications (e.g., removing a window by replacing it with wall content), adding blur, defining edges, etc.). Segments may be processed in phases executed according to timing constraints associated with the display of individual blended frames (e.g., start and end), such as blending with rendered content and / or being modified. This method involves displaying mixed frames, including effects.

[0005] Some implementations involve a method for achieving pass-through effects via shared programmable elements (e.g., GPUs), where the time division of the shared programmable elements is managed in a manner that prioritizes the execution of effects and / or blending over other processes (e.g., using a time-sharing strategy) to ensure that display-based timing constraints are met. In some implementations, an electronic device performs a method (e.g., via one or more processors). This method involves performing one or more steps or processes. In some implementations, the method involves acquiring image data comprising frames of images sequentially captured in a physical environment by an image capture device on an electronic device. The method may involve generating blended frames by sequentially processing segments of frames via shared programmable elements comprising a non-fixed instruction set, wherein the shared programmable elements are shared among multiple processes according to a time-sharing strategy. The multiple processes may include performing one or more visual effects on the segments, blending the segments with rendered content, and one or more additional processes (e.g., lighting adjustments, flicker suppression, rendering of game or application graphics, etc.). Time-sharing configuration controls the use of programmable elements by multiple processes to ensure that one or more visual effects and / or blending are performed according to timing constraints associated with the display of individual blended frames. The method involves displaying blended frames, including effects.

[0006] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform aspects of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0007] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0008] Figure 1 Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0009] Figure 2 It shows the implementation of some specific methods by Figure 1 The device provides a view of the XR environment.

[0010] Figure 3 This is a block diagram illustrating an exemplary captured image-rendered content combination pipeline according to some specific implementation, wherein pass-through effects are achieved by using programmable elements.

[0011] Figure 4 This is a block diagram illustrating another exemplary captured image-rendered content combination pipeline according to some specific implementation, wherein pass-through effects are achieved by using programmable elements.

[0012] Figure 5 This is a block diagram illustrating another exemplary captured image-rendered content combination pipeline according to some specific implementation, wherein pass-through effects are achieved by using programmable elements.

[0013] Figure 6 This is a flowchart illustrating an exemplary method, according to some specific implementation, for providing pass-through effects via the use of programmable elements in a captured image-rendered content combination pipeline.

[0014] Figure 7 This is a flowchart illustrating another exemplary method, according to some specific implementation, for providing pass-through effects by using programmable elements in a captured image-rendered content combination pipeline.

[0015] Figure 8 It is a block diagram based on some specific implementations of electronic devices.

[0016] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0017] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0018] Figure 1 An exemplary electronic device 105 operating in physical environment 100 is illustrated. Figure 1 In the example, physical environment 100 is a room with plants 120, a sofa 125, a first wall-mounted picture, a second wall-mounted picture 134, a third wall-mounted picture 136, and a table 135. Electronic device 105 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to provide pass-through video (e.g., to provide an XR environmental view) and / or capture and evaluate information about physical environment 100 and objects within it, as well as information about user 102 of electronic device 105. Information about physical environment 100 and / or user 102 can be used to provide visual and audio content, and / or identify the current location of physical environment 100 and / or the location of the user within physical environment 100.

[0019] In some implementations, a view of the XR environment may be provided to one or more participants (e.g., to user 102 via electronic device 105, and / or to other participants not shown via other electronic devices). Such an XR environment may be a 3D environment generated based on camera images and / or depth camera images of the physical environment 100. Such an XR environment may include virtual content (e.g., one or more virtual content items) positioned at one or more 3D locations relative to a 3D coordinate system associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100. In one example, the XR environment includes virtual content items displayed within a completely virtual surrounding environment. In one example, the XR environment includes virtual content items displayed within a completely real surrounding environment (e.g., combined with pass-through video provided by the device). In one example, the XR environment includes virtual content items displayed within an surrounding environment that includes both real and other virtual elements. In one example, the XR environment includes modifications to pass-through video of a real physical environment. In the XR environment view, the appearance of rendered virtual content and / or the surrounding environment can be configured or modified to provide effects and / or otherwise enhance the view or experience.

[0020] In some implementations, the XR environment is presented using pass-through video that depicts the physical environment (e.g., physical environment 100). For example, the pass-through video may be provided based on images received and presented from an image sensor (e.g., an outward-facing camera) of a device (e.g., device 105). In some implementations, virtual content items are presented along with the pass-through video content, and one or more effects are generated to alter the appearance of the virtual content items and / or the pass-through video of physical environment 100.

[0021] Figure 2 It shows the device (e.g., Figure 1 The device 105) provides an exemplary XR environment view 200. The XR environment view 200 includes the physical environment (e.g., Figure 1 The (live) pass-through video 205 of the physical environment 100, and the virtual content item 240 (e.g., movie, TV show, etc.) located at a 3D position within the XR environment.

[0022] The passthrough video 205 was also modified to provide visual effects. The passthrough video 205 includes a depiction 220 of a flower 120, a depiction 225 of a sofa 125, a depiction 230 of a wall 130, a depiction 232 of a first wall-mounted picture 132, a depiction 234 of a second wall-mounted picture 234, and a depiction 235 of a table 135, etc. Some or all of these depictions can be modified through effects (e.g., by changing pixels in the image of the passthrough video 205).

[0023] In this example, the pass-through video 205 is programmatically modified to reposition the depiction 232 of the first wall-mounted image 132. Specifically, the frame pixels of the pass-through video 205 corresponding to the first wall-mounted image 132 are shifted to the left (replacing the pass-through content that would originally be there). Similarly, the pass-through video 205 is programmatically modified to reposition the depiction 234 of the second wall-mounted image 134. Specifically, the frame pixels of the pass-through video 205 corresponding to the second wall-mounted image 134 are shifted to the left and down (replacing the pass-through content that could originally be there). The frame pixels corresponding to the third wall-mounted image 136 are completely removed, i.e., replaced with pixels corresponding to the depiction 230 of the virtual content item 240 and the wall 130. In this example, these modifications may be based on programming instructions that identify when virtual content items, such as virtual content item 240, will be positioned on the wall and obscure (or partially obscure) wall-mounted elements, and instructions that specify transparent modifications to move such obscure items to an unobscure location (i.e., as in the cases of depictions 232 and 234) or to remove such obscure items (i.e., as in the case of the depiction of the third wall-mounted image 136 (not shown) being removed).

[0024] The virtual content item 240 and modifications to the pass-through video 205 can be fixed over time (e.g., frame-to-frame) to provide a consistent view of the XR environment across multiple frames. Therefore, when a user moves device 105 around physical environment 100 to view the XR environment from different viewpoints, the virtual content item 240 can appear to remain in a fixed position, for example, world-locked to appear to remain in a fixed position on the wall of physical environment 100. Similarly, modifications to the pass-through can be consistently applied such that the depiction 232 of the first wall-mounted image 132 and the depiction 234 of the second wall-mounted image 134 remain fixedly positioned on the depiction 230 of wall 130.

[0025] The processor can be configured to process frames of passthrough video at a relatively high frame rate (FPS), such as, for example, a frame rate greater than 60 FPS. In some implementations, one or more processing components retrieve each frame or frame segment of passthrough video from an outward-facing camera and modify each passthrough frame or frame segment according to programming instructions (e.g., instructions to add, remove, reposition, enhance, or otherwise alter the appearance of real-world content) to achieve one or more effects. This process may include an alpha blending process for combining each frame of passthrough video (e.g., background content) with frame-specific virtual content. In some implementations, the alpha-blending value associated with the passthrough video in an area corresponding to the virtual content item and / or the surrounding environment can be adjusted. Blending and passthrough modifications can be performed by a single element of the display pipeline or by multiple elements, for example, sequentially.

[0026] Figure 3A process 300 including an exemplary pipeline is illustrated, wherein pass-through effects are implemented using programmable elements. The pipeline can combine captured image content with rendered content. In this example, the exemplary captured image-rendered content combining pipeline includes a rendering process 330 and a blending / effects process 335 implemented via one or more graphics processing units / programmable hardware 325 (e.g., executing code via such hardware).

[0027] exist Figure 3 In this example, one or more cameras 305 provide image 307 to image signal processor (ISP) 310. ISP 310 may perform one or more processes to control the capture of the content depicted in image 307 and / or change the appearance of that content (e.g., calibrating and modifying parameters such as white balance, exposure control, autofocus, noise reduction, sharpness adjustment, etc.). Image frame content 312 is provided to blending / effects process 335, which in this example is a combination unit (e.g., instruction set) executed within GPU / programmable hardware 325.

[0028] One or more virtual content sources 315 provide virtual content 317 (e.g., 2D or 3D objects to be rendered so that they appear at 3D locations within a 3D environment depicted by image frame content 312). The rendering process 330 may determine virtual frame content 332 based on the virtual content 317. Such virtual frame content 332 may be generated by rendering a 2D view of the virtual content from a specific 3D viewpoint. For example, image frame content 312 may correspond to a specific viewpoint within a 3D physical environment depicted therein, and rendering a view of the virtual content 317 may involve determining the 3D location of the virtual content within that 3D environment and then rendering a view of the virtual content 317 from the current camera viewpoint within that 3D environment. The rendering process 330 may produce virtual frame content 332, which is then provided to a blending / effects process 335.

[0029] The blending / effects process 335 combines virtual frame content 332 with image frame content 312 to produce a combined view, that is, a view depicting virtual content 317 at an appropriate 3D location within the 3D environment depicted in image frame content 312. The blending / effects process 335 may also execute programming instructions to apply modifications or other effects to image frame content 312 (e.g., pass-through video) and / or virtual frame content 332. Modifications or other effects may be applied before or after combining image frame content 312 with virtual frame content 332. The blended / modified frame content 337 is provided for display via display 350.

[0030] In some implementations, one or more processes are performed for rendering virtual content (e.g., rendering process 330), mixing the rendered virtual content with pass-through frames (mixing / effects process 335), and / or changing the appearance of the rendered virtual content and / or pass-through video content (mixing / effects process 335). One or more of these display pipeline functions or other features may be executed via fixed-function hardware that executes only fixed instructions (e.g., via an application-specific integrated circuit (ASIC) designed for a specific task). However, according to the specific implementations disclosed herein, one or more of these display pipeline functions or other features are executed using one or more processors (e.g., GPU / programmable hardware 325) capable of executing programming instructions to implement such functions. Some or all of these functions may be executed via processors (e.g., GPUs) capable of executing programming instructions to achieve various advantages (e.g., enabling pass-through modifications that would otherwise be impossible).

[0031] Some implementations enable direct manipulation of pass-through video (e.g., non-realistic rendering, blurring, stylization, distortion, displacement (e.g., shifting the depiction of a physical object 15 pixels to the right), depth-aware matting, per-surface-type optimized matting, etc.). Some implementations use and / or mix additional sensor data streams (e.g., data from other cameras on other devices in the device or environment). Some implementations enable applications to programm the pass-through effect and / or otherwise programmatically control the overall visual appearance of the pass-through video. In one example, an application can programmatically replace a portion of a user's wall with a Van Gogh painting by manipulating the pass-through pixels. In another example, a VR stained-glass window (e.g., through which real-world content is colored) can be provided by giving instructions to recolor, blur, distort, and / or otherwise change the pass-through color. In yet another example, a physical wall can be removed and replaced with image data from a camera behind a wall in a room, providing a "Superman" / "X-ray" visual effect.

[0032] Visual effects can take various forms. Some specific implementations offer passthrough spatial enhancement, non-photorealistic adjustments, accessibility effects, and so on. Visual effects can be added to a passthrough; for example, based on a computational analysis of the wall's appearance, notes can be added to a wall so that the notes appear to be a natural part of the wall. Visual effects can provide content to be added that matches the appearance of real-world elements depicted in other parts of the passthrough. Visual effects can be provided, for example, by applying contextual information based on conditions, incidents, user states, or other programmatically discernible information. Effects can provide artifact mitigation and / or compensation, for example, compensation for visual artifacts in camera image data. Effects can involve controlling hardware, for example, to reduce the source and / or occurrence of flicker. Effects can provide blending or tone mapping. Effects can involve effects achieved via direct rendering to the display backend (e.g., direct rendering to a warp / group space (explained below)) or otherwise integrated into the passthrough feed.

[0033] The specific implementations disclosed herein can be configured to achieve different types of visual effects. A first exemplary category of such effects (e.g., referred to herein as "Category 1" effects) may include effects that do not require information about previous frames or about distant pixels in the current frame (e.g., information about pixels that are more than a threshold number of pixels away or pixels associated with previously processed segments of the current frame), such as pixel shading or color changing. A second exemplary category of such effects (e.g., referred to herein as "Category 2" effects) may include effects that may require information about previous frames or about distant pixels (e.g., pixels that are more than a threshold number of pixels away or pixels associated with previously processed segments of the current frame), such as displacement, refraction effects, etc. The architecture can be customized according to the requirements of the effect to be achieved. For example, implementing Category 2 effects may require the use of additional components to store information (about distant portions of the previous or current frame) in memory.

[0034] Some specific implementations disclosed herein achieve the effect of event triggering. For example, changes to pass-through and / or rendered virtual content can be programmatically triggered based on the detection of user input or sensor data events, such as triggering events based on sensor data observed using sensor data obtained via a camera or other sensors on the device.

[0035] Some specific implementations disclosed herein achieve the same frame effect. For example, this may involve using an upstream model configured to observe the pass-through video stream and provide information (e.g., coefficients, instructions, etc.) for downstream post-processing applications (e.g., implementing keying). In some implementations, the same frame effect involves a pipeline that runs from camera to ISP to GPU to [optional fixed functions, e.g., blending] to display. This process can occur over a short time span (e.g., 11 ms) and can be pipelined, for example, with portions of frames being processed sequentially as they arrive from the camera. In some scenarios, it may be possible that the ISP process completes a large number of captured frames before the GPU completes the first / early portion of the frame. In parallel with the ISP to GPU path, this process can pipeline camera frame segments for processing (e.g., via a neural engine / computation). The neural engine can run a process / algorithm and generate parameters for either (a) the optional fixed functions between the GPU and the display; or (b) the GPU to be used at the end of its program. Such parameters can be used to identify (e.g., based on pixel values ​​at that stage of the pipeline) which pixels should be labeled, for example, labeled as "hand" pixels or non-hand pixels. The above can be applied to any algorithm of the same frame (e.g., for occlusion between real and virtual objects, such as XR adjustments that change physical environment walls into virtual calendars, etc.).

[0036] Some specific implementations provide or utilize deferred rendering. Unlike GPU rendering of virtual content pixels, this can involve rendering (e.g., rendering to a G-buffer) using descriptive geometric metadata with specified characteristics (e.g., diffuse color, reflectivity, normals, albedo, etc.). Pixels can be rendered by a display pipeline that applies lighting effects to both pass-through content and virtual content. Another example involves rendering improvements (e.g., subpixel-aware text rendering). Another example involves camera mitigation measures (e.g., resolving ghosting, glass glare, flicker, etc.).

[0037] In some implementations, a single System-on-a-Chip (SoC) provides the display pipeline.

[0038] Example of timing requirements One or more processes for rendering virtual content (e.g., rendering process 330), blending the rendered virtual content with pass-through frames (blending / effects process 335), and / or changing the appearance of the rendered virtual content and / or pass-through video content (blending / effects process 335) can be executed via a processor (e.g., a GPU) capable of executing programming instructions. Such processes can be configured to avoid various problems, such as timing issues, quality issues, etc. For example, providing pass-through video may require meeting timing requirements, such as ensuring that each frame or frame segment is fully processed in a timely manner for display. Using fixed-function hardware to blend the rendered virtual content with pass-through video can facilitate meeting such timing requirements. However, this pipeline can be inflexible in implementing programmed effects. Some implementations offer greater flexibility in implementing programmed effects using a processor (e.g., a GPU) capable of executing programming instructions to blend the rendered virtual content with pass-through frames and / or change the appearance of the rendered virtual content and / or pass-through video content. Because the program instructions are not fixed in nature, the time required for such processes before execution may be indeterminate. This uncertainty—the possibility that these processes will take more time than is available to meet display timing requirements—can be accounted for using additional processes (e.g., safeguards and / or rollbacks). Such additional processes (e.g., safeguards and / or rollbacks) can be configured to provide a display process that is flexible in both programming mix and effects, while also being compliant with display timing and / or other requirements. XR views can be configured with effects that were previously impossible, while also ensuring compliance with “live” pass-through timing requirements.

[0039] Some implementations provide a fixed photon-to-photon programmable display pipeline, i.e., a pipeline configured to ensure that fixed photon-to-photon delay requirements are met. Some implementations introduce flexible computation at various stages in the camera-to-display pipeline while preserving the fixed photon-to-photon delay. Some implementations provide pass-through beam tracking (i.e., meeting timing requirements), store one or more previously completed frames, and / or provide hardware post-projection of those past frames. Each new computation in the pipeline has access to the current portion of the current frame and one or more reprojected (e.g., considering device movement) previous frames.

[0040] Some specific implementations provide a computer architecture with programmable elements that implement pass-through in a beam-tracing-based captured image-rendered content composition pipeline. In this captured image-rendered content composition pipeline, a programmable block is implemented between the ISP (pass-through) and the fixed display pipeline hardware, allowing effects to be applied to the pass-through. In this pipeline, captured image data is processed (e.g., modified and combined with rendered content) in stages that must begin / end according to timing constraints to ensure timely display (i.e., live pass-through via beam tracing). This processing occurs in fewer than a full frame.

[0041] Figure 4 A process 400 including an exemplary pipeline is illustrated, wherein pass-through effects are implemented via the use of programmable elements. The pipeline can combine captured image content with rendered content. In this example, the exemplary captured image-rendered content combining pipeline includes a rendering process 430, a blending / effects process 435, and a post-blending process 440 implemented via a graphics processing unit / programmable hardware 425 (e.g., executing code via such hardware).

[0042] exist Figure 4 In this example, one or more cameras 405 provide image 407 to ISP 410. ISP 410 may perform one or more processes to control the capture of content in image 407 and / or change the appearance of that content (e.g., calibrating and modifying parameters such as white balance, exposure control, autofocus, noise reduction, sharpness adjustment, etc.). Image frame content 412 is provided to blending / effects process 435, which in this example is performed within GPU / programmable hardware 425.

[0043] One or more virtual content sources 415 provide virtual content 417 (e.g., a 2D or 3D object to be rendered so that it appears at a 3D location within a 3D environment depicted by image frame content 412). Rendering process 430 may determine virtual frame content 432 based on virtual content 417. Such virtual frame content 432 may be generated by rendering a 2D view of the virtual content from a specific 3D viewpoint. For example, image frame content 412 may correspond to a specific viewpoint within a 3D physical environment depicted therein, and rendering a view of virtual content 417 may involve determining the 3D location of the virtual content within that 3D environment and then rendering a view of virtual content 417 from the current camera viewpoint within that 3D environment. Rendering process 430 may produce virtual frame content 432, which is then provided to blending / effects process 435.

[0044] The blending / effects process 435 can execute programming instructions to combine virtual frame content 432 with image frame content 412 to produce combined frame content 436, that is, a view of virtual content 417 depicted at an appropriate 3D location within the 3D environment depicted in image frame content 412.

[0045] The blending / effects process 435 can also execute programming instructions to apply modifications or other effects to the combined frame content 436. The blending / effects process 435 can utilize information retrieved from memory 420 to achieve effects. Memory 420 can store information from previous frames or other portions of the current frame (e.g., output from previous iterations of a previous frame or portions of the current frame other than the currently being processed portion of the pipeline). In the case of previous frames, the stored information can be adapted (via information provided by ISP 410) to align with the viewpoint of the current frame. Information from memory 420 can facilitate more robust or otherwise improved edge detection, blurring, etc. Multi-channel effects can be implemented using information from memory 420. Some implementations are configured to reduce power consumption by minimizing or avoiding unnecessary transmission / buffering of image frames, for example, storing them only when frames or portions of frames will be needed.

[0046] The output 441 of the post-mixing process 440 includes mixed frames (or frame portions) with effects applied, which are displayed at the display 450 via the display pipeline 445 (implemented in fixed-function hardware 442).

[0047] Process 400 considers scenarios or situations where the blending / effects process 435 (or other processes implemented via programmable hardware 525) may or may not complete in a timely manner to meet display timing requirements. In this example, a fail-safe path 460 is provided directly from the ISP 410 to the display pipeline 445, so that if the output from the post-blending process 440 cannot be used for display in a timely manner, the image frame content 412 can still be displayed. Therefore, in such cases, the image frame content 412 will be displayed, thus avoiding the display of blank or corrupted content.

[0048] Additionally or alternatively, some specific embodiments disclosed herein utilize constraints to ensure that the effects provided by using programmable blocks do not degrade the performance of a beam-tracing-based captured image-rendered content composition pipeline. In a captured image-rendered content composition pipeline, constraints or other processes may be implemented to ensure timely display (i.e., live passthrough via beam tracing) of captured image data in stages that must begin / end according to timing constraints (e.g., being modified and combined with rendered content). For example, applications may be able to programmatically specify changes to passthrough video and / or virtual content. However, such procedures may be checked and constrained prior to execution (e.g., to limit the number of instructions allowed to be executed and / or the amount of processing). Doing so avoids or at least reduces the chance of programmable block failures. In another example, time constraints and / or other processing constraints may be employed in real-time during execution to abort or prevent operations that would violate time constraints. In another example, event-triggered mitigation measures may be employed. In yet another example, memory budget constraints may be employed.

[0049] Example of gaze point / group spatial processing This can involve foveated (also known as “group”) space processing, where programmable blocks (e.g., capable of executing code) act on and / or blend both passthrough content and rendered content in a non-linear space (e.g., foveated / group space). This can also involve primitive (i.e., non-foveated / group) space processing, where passthrough video is changed to its original form (or another form not adjusted for foveated) via programmable blocks. Some implementations provide a fallback pipeline that allows the display of passthrough video in the event of a programmable block failure.

[0050] Figure 5 A process 500 including an exemplary pipeline is illustrated, wherein pass-through effects are achieved via the use of programmable elements. The pipeline can combine captured image content with rendered content. In this example, the exemplary captured image-rendered content combining pipeline includes a rendering process 530, a warp grouping process 534, a blending process 535, and a post-blending process 540, implemented via a graphics processing unit / programmable hardware 525 (e.g., executing code via such hardware).

[0051] exist Figure 5In this example, one or more cameras 505 provide images 507. These images 507 are processed via fixed-function hardware 511, including an ISP 510 and a warp group process 515. The ISP 510 can perform one or more processes to control the capture of content in the image 507 and / or change the appearance of that content (e.g., calibrating and modifying parameters such as white balance, exposure control, autofocus, noise reduction, sharpness adjustment, etc.). The warp group process 515 converts the image from a modified format (e.g., where the pixel density is constant throughout the frame) to a warped or group-space format, where the pixel density is greater in a portion of the frame corresponding to the area around the user's viewing area (e.g., the foveated region) than in other areas (e.g., non-foveated regions). This simplifies the operations performed on the image (and thus provides improved efficiency) while preserving accuracy / detail in the foveated region. Both undistorted / ungrouped image frame content 512 and / or distorted / grouped image frame content 513 can be provided to the blending process 535, which in this example is performed within the GPU / programmable hardware 525.

[0052] One or more virtual content sources 515 provide virtual content 517 (e.g., a 2D or 3D object to be rendered so that it appears at a 3D location within a 3D environment depicted by image frame content). The rendering process 530 may determine virtual frame content 532 based on the virtual content 517. Such virtual frame content 532 may be generated by rendering a 2D view of the virtual content from a specific 3D viewpoint. For example, undistorted / ungrouped image frame content 512 may correspond to a specific viewpoint within the 3D physical environment depicted therein, and rendering a view of the virtual content 517 may involve determining the 3D location of the virtual content within that 3D environment and then rendering a view of the virtual content 517 from the current camera viewpoint within that 3D environment.

[0053] Rendering process 530 may produce virtual frame content 432, which is provided to warp group process 533, which in this example executes within GPU / programmable hardware 525. Warp group process 533 converts the image from a modified format (e.g., where the pixel density is constant throughout the frame) to a warp or group space format, where the pixel density is greater in the portion of the frame corresponding to the area around the user's viewing area (e.g., the foveated region) than in other areas (e.g., non-foveated regions). This simplifies the operations performed on the image (and thus provides improved efficiency) while preserving accuracy / detail in the foveated region. Warp group processes 515 and 533 may be configured to use a common output format to facilitate blending by blending / effects process 535.

[0054] The blending / effects process 535 can also execute programming instructions to combine the distorted / grouped virtual frame content 532 with the distorted / grouped image frame content 513 to produce a combined frame content 536, that is, a view of the virtual content depicted at an appropriate 3D location within the 3D environment depicted in the image frame content.

[0055] The blending / effects process 535 can also execute programming instructions to apply modifications or other effects to the combined frame content 536. The blending / effects process 535 can utilize information retrieved from memory 520 to achieve the effects. Memory 520 can store information from previous frames or other portions of the current frame (e.g., output from previous iterations of a pipeline from a previous frame or from portions of the current frame other than the currently being processed portion). In the case of a previous frame, the stored information can be adapted (via information provided by ISP 510) to align with the viewpoint of the current frame. The blending / effects process 535 can utilize undistorted / ungrouped image frame content 512.

[0056] The output 541 of the post-mixing process 450 includes mixed frames (or frame portions) with effects applied, which are displayed at the display 555 via the display pipeline 550 (implemented in fixed-function hardware 542).

[0057] Process 500 considers scenarios or situations where the blending / effects process 535 (or other processes implemented via programmable hardware 525) may or may not complete in a timely manner to meet display timing requirements. In this example, a fail-safe path 560 is provided directly from the ISP 510 or the warp grouping process 515 to the display pipeline 550, so that if the output 541 cannot be used for display in a timely manner, the ungrouped image frame content 512 or the grouped image frame content 513 can be used for display. Therefore, in such cases, the ungrouped image frame content 512 or the grouped image frame content 513 will be used to provide the current frame view, thereby avoiding the display of blank or corrupted content in such cases.

[0058] Some of the specific implementations disclosed herein improve upon existing pipeline techniques, such as techniques for combining virtual content with camera content using fixed-function hardware implementations. In some existing systems, the GPU is unavailable or configured to render effects in warp / group space. The conversion to warp / group space may occur only on fixed-function hardware, i.e., thus requiring the GPU to render its output in group space. The specific implementations disclosed herein deliver a warp / group space version of the camera image to the GPU for processing, e.g., to apply effects. Blending may additionally occur in warp / group space. In some implementations, the system is configured to alternate between different modes, such as a first mode applying effects in warp / group space and a second mode applying effects without conversion to warp / group space. The mode may be changed based on circumstances, e.g., utilizing the second mode when more resources and / or time are available.

[0059] In various specific implementations, a processing component capable of executing software-defined instructions is configured between the pass-through video acquisition component and the display system component. This can be achieved in various ways, as well as in ways that provide efficiency and other benefits. Figures 3 to 5 The example provided is for illustration. Other specific implementations are envisioned.

[0060] In some implementations, the GPU used in the display pipeline shares functionality with other features. For example, such a GPU may additionally perform operations performed by applications running on the device (e.g., game engines). Sharing may involve partitioning GPU time. This sharing can be achieved through a process of enforcing rules to ensure that display timing requirements are met.

[0061] In various specific implementations, the pipeline may additionally include lens correction distortion (e.g., lens geometry distortion correction, time distortion, post-distortion, etc.), matting distortion / blending, panel correction or compensation (lens, panel, or both), lighting effects, flicker suppression, and / or other processes to improve the appearance of the displayed content.

[0062] Example Method Figure 6 This is a flowchart illustrating an exemplary method 600 for providing a pass-through effect via the use of programmable elements in a captured image-rendered content composition pipeline. In some embodiments, method 600 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some embodiments, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as... Figure 1(Device 105). In some embodiments, method 600 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 600 is executed by one or more processors that execute code.

[0063] At block 602, method 600 relates to obtaining image data comprising frames of images captured sequentially in a physical environment by an image capture device on an electronic device.

[0064] At block 604, method 600 relates to generating blended frames by sequentially processing fragments of frames via a programmable element (e.g., a GPU). Each fragment comprises fewer than a full frame (e.g., small fragments facilitate beam tracing). The programmable element includes a non-fixed set of instructions, wherein processing includes performing one or more visual effects on the fragments and blending the fragments with rendered content. The fragments are modified with one or more visual effects and blended with rendered content during phases executed according to timing constraints associated with the display of individual blended frames (e.g., start and end).

[0065] At box 604, method 600 involves displaying a mixed frame.

[0066] Some implementations utilize warp / group space (i.e., corresponding to foveated and non-foveated regions). For example, programmable blocks can act on and / or blend both passthrough and rendering in nonlinear space (e.g., foveated / group space). In method 600, a frame can be converted to a nonlinear format based on an identifier of the region of the frame that the user is viewing, and one or more visual effects can be performed via programmable elements based on the nonlinear format. A frame can be converted to a nonlinear format based on an identifier of the region of the frame that the user is viewing, and blending can be performed via programmable elements based on the nonlinear format.

[0067] Some specific implementations involve using raw space (i.e., not warped / grouped space) to achieve pass-through effects via programmable blocks. In method 600, one or more visual effects and / or blending can be performed via programmable elements based on the raw linear format of the frame.

[0068] Some implementations provide a fallback process for fault conditions, i.e., a fallback pipeline that allows for display pass-through in the event of a programmable block failure. Method 600 may involve: detecting a condition in which the programmable block cannot perform one or more visual effects or blending according to timing constraints in timing constraints; and based on the detection of this condition, providing one or more frames for display without processing via programmable elements.

[0069] In some implementations, the programmable element is a shared GPU, and other processes are selectively interrupted to account for failure conditions; that is, a fallback pipeline is provided to allow display pass-through in the event of a programmable block failure. In method 600, the programmable element may be a GPU. This GPU may be shared among multiple processes, including processes for performing visual effects, blending, and additional processes. Exemplary additional processes may involve providing lighting control / effects, flicker suppression, rendering of game or application graphics, etc. Method 600 may involve: detecting a condition in which the GPU cannot perform one or more visual effects or blending according to timing constraints; and, based on the detection of this condition, modifying the additional processes to enable the GPU to perform one or more visual effects or blending according to timing constraints.

[0070] Some specific implementations achieve Category 1 effects, i.e., effects that do not require information from previous frames or distant pixels, such as changing pixel color based on pixels that are more than a threshold number of pixels away or pixels associated with previously processed segments in the current frame. In method 600, performing one or more visual effects on a segment may involve segment-specific effects, where each effect is based solely on information about the pixels in the corresponding segment to which the corresponding segment is performed. One or more visual effects may, for example, involve changing pixel color, i.e., without referencing the color or state of any other pixels.

[0071] Some specific implementations achieve Category 2 effects, i.e., effects that require knowledge of previous frames or distant pixels, such as displacement using additional components (e.g., previous frames, memory). In method 600, performing one or more visual effects may involve a multi-segment effect, where the effect for a first segment is based on information about one or more pixels of a second segment from the same frame that is different from the first segment. In one example, the effect involves displacement of frame content from the second segment to the first segment. Performing one or more visual effects may involve a multi-segment effect, where the effect for a first segment is based on information about one or more pixels of a second segment from a previous frame.

[0072] Some specific implementations achieve event-triggered programmable pass-through effects, for example, programmable / GPU / ANE-based processing triggered by camera events. In method 600, one or more visual effects may involve at least one visual effect triggered based on the detection of an event, wherein the event is detected based on the evaluation of at least one frame in the frame.

[0073] Some specific implementations achieve, for example, the same frame effect using an upstream model that observes the pass-through flow and provides coefficients for downstream later applications (e.g., hand-cutting). In method 600, one or more visual effects may involve triggering at least one visual effect in the current frame based on analysis of previous segments of the current frame.

[0074] In some implementations, the timing constraints of method 600 are based on the timing requirement of providing live pass-through at a predetermined frame rate.

[0075] In some implementations, in method 600, fragments are mixed with rendered content to provide a view of the XR environment. In some implementations, the electronic device of method 600 is an HMD (Head-Down Device).

[0076] Figure 7 This is a flowchart illustrating another exemplary method for providing pass-through effects via the use of programmable elements in a captured image-rendered content composition pipeline. In some embodiments, method 700 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some embodiments, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as... Figure 1 (Device 105). In some embodiments, method 700 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 700 is executed by one or more processors executing code.

[0077] At block 702, method 700 relates to obtaining image data comprising frames of images captured sequentially in a physical environment by an image capture device on an electronic device.

[0078] At block 704, method 700 relates to generating a blended frame by sequentially processing fragments of a frame via a shared programmable element comprising a non-fixed instruction set. This shared programmable element is shared among multiple processes according to a time-sharing strategy. The multiple processes include performing one or more visual effects on the fragments, blending the fragments with rendered content, and one or more additional processes (e.g., lighting, flicker suppression, rendering of game or application graphics, etc.). The time-sharing configuration controls the use of the programmable element by the multiple processes to ensure that one or more visual effects or blends are performed according to timing constraints associated with the display of the individual blended frames.

[0079] At box 704, method 700 involves displaying a mixed frame.

[0080] In some implementations, the details of controlling the shared programmable element are specified via a time-sharing strategy. This shared programmable element is a shared GPU.

[0081] In some implementations, method 700 involves: detecting a condition where the shared GPU cannot execute one or more visual effects or blendings according to timing constraints; and, based on detecting the condition and the time-sharing strategy, modifying one or more additional processes to enable the GPU to execute one or more visual effects or blendings according to timing constraints. Modifying one or more additional processes may involve interrupting the processing of one or more processes via the shared GPU.

[0082] Some implementations utilize warp / group space. In method 700, a frame can be converted to a non-linear format based on an identifier of the region of the frame that the user is viewing, and one or more visual effects can be performed via programmable elements based on the non-linear format. A frame can be converted to a non-linear format based on an identifier of the region of the frame that the user is viewing, and blending can be performed via programmable elements based on the non-linear format.

[0083] Some implementations utilize the original space (e.g., not a warp / group space). In method 700, one or more visual effects and blending can be performed via programmable elements based on the original linear format of the frame.

[0084] Some implementations provide fallback processing for fault conditions, i.e., provide a fallback pipeline that allows display pass-through in the event of a programmable block failure. Method 700 may involve: detecting a condition in which the programmable block cannot perform one or more visual effects or blending according to timing constraints in timing constraints; and based on the detection of this condition, providing one or more frames in a frame for display without processing via programmable elements.

[0085] Some specific implementations achieve Category 1 effects, i.e., effects that do not require information about pixels from previous frames or pixels located more than a threshold number of pixels away, or pixels associated with previously processed segments in the current frame; for example, changing pixel color. In method 700, one or more visual effects on a segment may involve segment-specific effects, where each effect is based solely on information about the pixels in the corresponding segment to which the corresponding segment was performed. One or more visual effects may involve changing pixel color, for example, without referencing another pixel.

[0086] Some specific implementations achieve Category 2 effects, such as effects that do require knowledge of previous frames or distant pixels, for example, by using displacement of additional components (e.g., previous frames, memory). In method 700, performing one or more visual effects may involve a multi-segment effect, where the effect for the first segment is based on information about one or more pixels of a second segment from the same frame that is different from the first segment. This effect may involve displacement of frame content from the second segment to the first segment. Performing one or more visual effects may involve a multi-segment effect, where the effect for the first segment is based on information about one or more pixels of a second segment from a previous frame.

[0087] Some specific implementations implement event-triggered pass-through effects programming, i.e., processing based on programmable / GPU / ANE based on camera events. In method 700, one or more visual effects include at least one visual effect triggered based on the detection of an event, wherein the event is detected based on the evaluation of at least one frame in the frame.

[0088] Some specific implementations achieve the same frame effect, i.e., the same frame effect via an upstream model that observes the pass-through flow and provides coefficients for downstream later applications (e.g., hand-cutting). In method 700, one or more visual effects may involve triggering at least one visual effect in the current frame based on analysis of previous segments of the current frame.

[0089] In some specific implementations, the timing constraints of method 700 are based on the timing requirement of providing live pass-through at a predetermined frame rate.

[0090] In some implementations, in method 700, fragments are mixed with rendered content to provide a view of the XR environment. In some implementations, the electronic device of method 700 is an HMD (Head-Down Display).

[0091] Figure 8 This is a block diagram of example device 800. Device 800 illustrates... Figure 1 An exemplary device configuration of electronic device 105. Although certain specific features have been illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 800 includes one or more processing units 802 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 804, one or more communication interfaces 808 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.14x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 810, output devices (e.g., one or more displays) 812, one or more internal and / or external image sensor systems 814, memory 820, and one or more communication buses 804 for interconnecting these components and various other components.

[0092] In some embodiments, one or more communication buses 804 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 806 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), one or more cameras (e.g., an inward-facing camera and an outward-facing camera of an HMD), one or more infrared sensors, one or more thermal sensors, etc.

[0093] In some embodiments, one or more displays 812 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc., to a user. In some embodiments, one or more displays 812 are configured to present content to a user (determined based on the user's determined user / object position within the physical environment). In some embodiments, one or more displays 812 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 812 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 800 includes a single display. In another example, device 800 includes displays for each of the user's eyes.

[0094] In some embodiments, one or more image sensor systems 814 are configured to acquire image data corresponding to at least a portion of the physical environment 100. For example, one or more image sensor systems 814 include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, one or more image sensor systems 814 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 814 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0095] In some specific implementations, sensor data can be obtained from devices (e.g., Figure 1The devices 105 and 110 acquire the sensor data during a scan of the room's physical environment. The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to views of the room captured during the scan. In some embodiments, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., depth images from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., accelerometers, gyroscopes, IMUs, etc.). In some embodiments, the sensor data includes visual inertial odometry (VIO) data determined based on the image data. The 3D point cloud can provide semantic information about one or more elements of the room. The 3D point cloud can provide information about the location and appearance of surface portions within the physical environment. In some embodiments, the 3D point cloud is acquired over time (e.g., during a scan of the room) and can be updated, with updated versions of the 3D point cloud obtained over time. For example, when the 3D representation is updated / adjusted over time (e.g., when a user scans a room), the 3D representation can be obtained (and analyzed / processed).

[0096] In some embodiments, sensor data may be positioning information, and some embodiments include VIO (Vehicle Identification and Odometry) to determine equivalent odometry information to estimate travel distance using sequential camera images (e.g., light intensity image data) and motion data (e.g., acquired from an IMU / motion sensor). Alternatively, some embodiments of this disclosure may include a Simultaneous Localization and Mapping (SLAM) system (e.g., a positioning sensor). This SLAM system may include a GPS-independent, multi-dimensional (e.g., 3D) laser scanning and range measurement system that provides real-time simultaneous localization and mapping. This SLAM system can generate and manage highly accurate point cloud data produced by reflections from laser scans of objects in the environment. Accurately tracking the movement of any points in the point cloud over time allows the SLAM system to use points in the point cloud as reference points for its location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0097] In some embodiments, device 800 includes an eye-tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes. Furthermore, the illumination source of device 800 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on a near-eye display of device 800.

[0098] Memory 820 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 820 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 820 optionally includes one or more storage devices remotely located to one or more processing units 802. Memory 820 includes a non-transitory computer-readable storage medium.

[0099] In some embodiments, memory 820 or a non-transitory computer-readable storage medium of memory 820 stores an optional operating system 830 and one or more instruction sets 840. Operating system 830 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 840 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 840 is software executable by one or more processing units 802 to implement one or more of the techniques described herein.

[0100] Instruction set 840 may include fixed and / or programmable instructions executable via fixed-function hardware and / or programmable components (such as a CPU or GPU) or a combination thereof to perform one or more of the features described herein, such as generating virtual content, rendering, distorting, blending, effects, performing other aspects of the display pipeline, etc. Instruction set 840 may be embodied in various forms as hardwired instructions, a single software executable, multiple software executables, etc.

[0101] Although instruction set 840 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside on a separate computing device. Furthermore, Figure 8 This is intended more as a functional description of various features present in a particular implementation than as a structural diagram of the specific implementation described herein. As will be appreciated by those skilled in the art, the items shown individually can be combined, and some items can be separated. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0102] Those skilled in the art will understand that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure more relevant aspects of the specific embodiments of the examples described herein. Furthermore, other effective aspects and / or variations do not include all the details in the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings only illustrate some exemplary embodiments of this disclosure and should not be considered limiting.

[0103] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0104] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all illustrated operations to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the partitioning of the various system components in the above embodiments should not be construed as requiring such partitioning in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0105] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0106] The embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0107] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmitting devices, or display devices of a computing platform.

[0108] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0109] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0110] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Furthermore, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0111] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0112] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” as used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0113] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method, the method comprising: At electronic devices: Obtain image data, which includes frames of images captured sequentially in a physical environment by an image capture device on the electronic device; A blended frame is generated by sequentially processing fragments of the frame via programmable elements, each fragment comprising less than a full frame of the frame, the programmable elements comprising a non-fixed instruction set, wherein the processing includes applying one or more visual effects to the fragments and blending the fragments with rendered content, wherein the fragments are modified with the one or more visual effects and blended with the rendered content in a phase executed according to timing constraints associated with the display of individual blended frames; and The mixed frame is displayed.

2. The method of claim 1, wherein the frame is converted to a non-linear format based on an identifier of the area of ​​the frame that the user is viewing, and the one or more visual effects are performed via the programmable element based on the non-linear format.

3. The method of claim 1, wherein the frame is converted to a non-linear format based on an identifier of the area of ​​the frame that the user is viewing, and the mixing is performed via the programmable element based on the non-linear format.

4. The method of claim 1, wherein the one or more visual effects and the blending are performed via the programmable element based on the original linear format of the frame.

5. The method according to claim 1, further comprising: The programmable block is detected as being unable to execute one or more visual effects or the mixture according to the timing constraints in the timing constraints; as well as Based on the detected condition, one or more frames from the frame are provided for display without processing via the programmable element.

6. The method of claim 1, wherein the programmable element is a graphics processing unit (GPU).

7. The method of claim 6, wherein the GPU is shared among a plurality of processes, the plurality of processes including processes for performing the visual effects, the blending and attaching processes.

8. The method according to claim 7, further comprising: The system detects a situation where the GPU cannot execute one or more visual effects or the blending according to the timing constraints in the timing constraints. as well as Based on the detected condition, the additional process is modified so that the GPU can execute the one or more visual effects or the blending according to the timing constraints.

9. The method of claim 1, wherein performing the one or more visual effects on the segment involves segment-specific effects, wherein each effect is based solely on information about the pixels in the corresponding segment to which the corresponding segment is applied.

10. The method of claim 9, wherein the one or more visual effects include changing pixel color.

11. The method of claim 1, wherein performing the one or more visual effects involves a multi-segment effect, wherein the effect for the first segment is based on information about one or more pixels of a second segment from the same frame that is different from the first segment.

12. The method of claim 11, wherein the effect includes displacement of frame content from the second segment to the first segment.

13. The method of claim 1, wherein performing the one or more visual effects involves a multi-segment effect, wherein the effect for the first segment is based on information about one or more pixels from a second segment of a previous frame.

14. The method of claim 1, wherein the one or more visual effects include at least one visual effect triggered based on the detection of an event, wherein the event is detected based on the evaluation of at least one of the frames.

15. The method of claim 1, wherein the one or more visual effects include triggering at least one visual effect in the current frame based on analysis of previous segments of the current frame.

16. The method of claim 1, wherein the timing constraint corresponds to providing live pass-through.

17. The method of claim 1, wherein the fragment is blended with rendered content to provide a view of an extended reality (XR) environment.

18. The method of claim 1, wherein the electronic device is a head-mounted device (HMD).

19. A system comprising: One or more processors and program instructions, wherein the program instructions, when executed on the one or more processors, cause the one or more processors to perform operations including the following: Obtain image data, which includes frames of images captured sequentially in a physical environment by an image capture device on the electronic device; A blended frame is generated by sequentially processing fragments of the frame via programmable elements, each fragment comprising less than a full frame of the frame, the programmable elements comprising a non-fixed instruction set, wherein the processing includes applying one or more visual effects to the fragments and blending the fragments with rendered content, wherein the fragments are modified with the one or more visual effects and blended with the rendered content in a phase executed according to timing constraints associated with the display of individual blended frames; and The mixed frame is displayed.

20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations including: Obtain image data, which includes frames of images captured sequentially in a physical environment by an image capture device on the electronic device; A blended frame is generated by sequentially processing fragments of the frame via programmable elements, each fragment comprising less than a full frame of the frame, the programmable elements comprising a non-fixed instruction set, wherein the processing includes applying one or more visual effects to the fragments and blending the fragments with rendered content, wherein the fragments are modified with the one or more visual effects and blended with the rendered content in a phase executed according to timing constraints associated with the display of individual blended frames; and The mixed frame is displayed.

Images