Synchronization circuit for reducing time delay associated with image transparent transmission

Abstract

In some implementations, a method of synchronizing content generation and delivery architectures to reduce latency associated with image pass-through. The method includes determining a time offset associated with the content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture; obtaining a first reference rate associated with a portion of the content generation and delivery architecture; generating, via a synchronization circuit, a synchronization signal for the content generation and delivery architecture based at least in part on the first reference rate; and operating the content generation and delivery architecture according to the synchronization signal and the time offset.

Description

Technical Field

[0001] The present disclosure relates generally to image processing, and in particular to systems, methods, and devices for reducing latency associated with image pass-through. Background Art

[0002] Typically, the content generation and delivery architecture for image passthrough includes multiple parts, such as an image capture device, an image signal processing (ISP) pipeline, multiple frame buffers, a display pipeline, and a display device. However, the content generation and delivery architecture can be associated with significant latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] In order that the present disclosure may be understood by those skilled in the art, a more particular description will be given with reference to some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0004] Figure 1A is a block diagram of an example content generation and delivery architecture according to some specific implementations.

[0005] Figure 1B It is implemented according to some specific Figure 1A Block diagram of the first part of the content generation and delivery architecture in .

[0006] Figure 1C It is implemented according to some specific Figure 1A Block diagram of the second part of the content generation and delivery architecture in .

[0007] Figure 2A Examples of some specific implementations and Figure 1A Example latency graph associated with the content generation and delivery architecture in .

[0008] Figure 2B According to some specific implementation Figure 1A Block diagram of an example partial frame buffer associated with a content generation and delivery architecture in FIG.

[0009] Figure 3A is a block diagram of an example content generation and delivery architecture with multiple image capture devices according to some implementations.

[0010] Figure 3B is a block diagram of another example content generation and delivery architecture with multiple image capture devices according to some implementations.

[0011] Figure 3C is a block diagram of an example content generation and delivery architecture with multiple display devices according to some implementations.

[0012] Figure 3Dis a block diagram of another example content generation and delivery architecture with multiple display devices according to some implementations.

[0013] Figure 3E is a block diagram of an example content generation and delivery architecture with multiple image capture devices and multiple display devices according to some implementations.

[0014] Figure 4A An example display-related interval sequence associated with transitioning from a first frame rate to a second frame rate is illustrated according to some implementations.

[0015] Figure 4B exemplifies the beginning of a sequence of image capture intervals relative to Figure 4A The in shows example time offsets between the beginnings of the correlated interval sequences.

[0016] FIG. 5A to FIG. 5C Various timing diagrams with different time offsets according to some implementations are illustrated.

[0017] Figure 6 is a flowchart representation of a method for synchronizing content generation and delivery architecture to reduce latency associated with image pass-through according to some specific implementations.

[0018] Figure 7 is a block diagram of an example computing system according to some implementations.

[0019] Fig. 8A is a flowchart representation of a method for synchronizing content generation and delivery architecture to reduce latency associated with image pass-through according to some specific implementations.

[0020] Figure 8B and Figure 8C A flowchart representation of a method for synchronizing content generation and delivery architecture to reduce latency associated with image pass-through according to some implementations is illustrated.

[0021] Fig. 9 Block diagrams of example controllers and example electronic devices are illustrated according to some implementations.

[0022] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0023] Various embodiments disclosed herein include devices, systems, and methods for synchronizing a content generation and delivery architecture to reduce latency associated with image pass-through. According to some embodiments, the method is performed at a computing system that includes a non-volatile memory and one or more processors, a synchronization circuit, and a content generation and delivery architecture that includes an image capture device, an image signal processing (ISP) pipeline, a portion of a frame buffer, a display pipeline, and a display device. The method includes: determining a time offset associated with the content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture; obtaining a first reference rate associated with a portion of the content generation and delivery architecture; generating a synchronization signal for the content generation and delivery architecture based at least in part on the first reference rate via the synchronization circuit; and operating the content generation and delivery architecture based on the synchronization signal and the time offset.

[0024] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for performing or causing the execution of any of the methods described herein.

[0025] According to some specific implementations, a computing system includes one or more processors, non-volatile memory, an interface for communicating with a display device and one or more input devices, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of a computing system having an interface for communicating with a display device and one or more input devices, cause the computing system to perform or cause the execution of the operations of any of the methods described herein. According to some specific implementations, a computing system includes one or more processors, non-volatile memory, an interface for communicating with a display device and one or more input devices, and a component for performing or causing the execution of the operations of any of the methods described herein. DETAILED DESCRIPTION

[0026] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0027] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system may detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to a representation of physical movement (e.g., a voice command).

[0028] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, μLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0029] As described above, the content generation and delivery architecture for image pass-through includes multiple parts, such as an image capture device, an image signal processing (ISP) pipeline, multiple frame buffers, a display pipeline, and a display device. As an example, the above-mentioned XR environment includes image or video pass-through, whereby the XR content is synthesized with or superimposed on the image or video of the physical environment. However, the content generation and delivery architecture may be associated with significant latency. In order to achieve lower latency for image pass-through, the signal path from the image capture device to the display device can be simplified by synchronizing clocks among multiple parts of the content generation and delivery architecture and completely removing multiple frame buffers or replacing multiple frame buffers with partial frame buffers.

[0030] To this end, a master sync generator (MSG) generates synchronization signals to synchronize clocks on a content generation and delivery architecture based on a reference rate (e.g., a refresh or display rate of a display device) and a time offset associated with a reduction in photon-to-photon latency (e.g., a duration between an image capture device capturing an image and a display device presenting the image). In some implementations, partial frame buffers are dynamically allocated based at least in part on the reference rate. According to some implementations, partial frame buffers may be removed. In some implementations, the content generation and delivery architecture determines / selects a time offset in order to reduce the photon-to-photon latency and also reduce the size of the partial frame buffer without exhausting the partial frame buffer, as described below with reference to FIG. 5A to FIG. 5C discussed.

[0031] Figure 1A is a block diagram of an example content generation and delivery architecture 100 according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the content generation and delivery architecture 100 includes a master sync generator (MSG) 120, an image capture device 142, an image signal processing (ISP) pipeline 144, a portion of a frame buffer 145, a display pipeline 146, and a display device 148. In various implementations, Figure 7 The computing system 700 in the embodiment includes the content generation and delivery architecture 100 or provides its functionality. According to some specific implementations, Fig. 9 The electronic device 900, Fig. 9 The controller 950, or a suitable combination thereof, includes the content generation and delivery architecture 100 or provides its functionality.

[0032] like Figure 1A As shown, the image capture device 142 captures an input image of the physical environment 105. The ISP pipeline 144 generates a processed image by performing one or more image processing operations on the input image and writes the processed image to the partial frame buffer 145. Figure 1B The image capture device 142 and the ISP pipeline 144 are described in more detail. Figure 2B The partial frame buffer is described in more detail. According to some implementations, the ISP pipeline 144 can be directly coupled to the display pipeline 146 without the intervening partial frame buffer 145.

[0033] like Figure 1AAs shown, the display pipeline 146 reads the processed image from the portion of the frame buffer 145 and generates a composite image by compositing the processed image frame with the rendered virtual content. Subsequently, the display device 148 presents the composite image or other content. Figure 1C Display pipeline 146 and display device 148 are described in more detail.

[0034] like Figure 1A As shown, MSG 120 obtains (eg, receives, retrieves, determines, selects, etc.) time offset 112 to reduce the time offset 112 referred to below. Figure 2A The described photon-to-photon delay 210 also reduces the memory footprint of the portion of the frame buffer 145. Figure 2A The time offset 112 is described in more detail. Figure 1A In the example, MSG 120 also obtains (eg, receives, retrieves, generates, etc.) a reference rate 114, such as a refresh rate or frame rate of a display device 148.

[0035] exist Figure 1A , MSG 120 generates a synchronization signal 132 based on time offset 112 and reference rate 114. MSG 120 provides synchronization signal 132 to image capture device 142 to synchronize its clock with display pipeline 146 based on reference rate 114. For example, synchronization signal 132 includes one or more values ​​of adjustable timing-related parameters associated with image capture device 142, such as shutter speed, exposure length, etc.

[0036] exist Figure 1A , the MSG 120 generates a synchronization signal 136 based on the time offset 112 and the reference rate 114. The MSG 120 provides the synchronization signal 136 to the display pipeline 146 to synchronize its clock with the image capture device 142 based on the reference rate 114. For example, the synchronization signal 136 includes one or more values ​​of adjustable timing-related parameters associated with the display pipeline 146 (such as one or more rendering operations performed by the display pipeline 146, a read operation performed by the display pipeline 146 on a processed image from the portion of the frame buffer 145, etc.).

[0037] In some implementations, synchronization signals 132 and 136 correspond to the same signal that includes time offset 112 and reference rate 114. In some implementations, synchronization signals 132 and 136 also include values ​​of one or more adjustable timing-related parameters of image capture device 142 and display pipeline 146, respectively.

[0038] In some implementations, the MSG 120 or a component thereof (e.g., the transition logic 122) detects a change (or variation) in the reference rate 114 from a first reference rate to a second reference rate. As one example, the reference rate 114 increases from 90 Hz to 100 Hz. As another example, the reference rate 114 decreases from 100 Hz to 90 Hz. One of ordinary skill in the art will appreciate that the above reference rates are arbitrary, non-limiting frequency values, and in various other implementations, these frequency values ​​may be replaced with numerous other frequency values.

[0039] For example, the reference rate 114 may change when the application changes, for example, from a productivity application to a content consumption application, etc. In this example, the application may change due to a selection or input from the user. For example, when the state of the user changes, for example, from sitting to standing, etc., the reference rate 114 may change. For another example, when the state of the environment changes, for example, from a bright environment to a dim environment, from a low-frequency environment to a high-frequency environment, etc., the reference rate 114 may change. For another example, when the presented content changes from a first content associated with a first region (e.g., a 24Hz or 60Hz playback rate of U.S. content) to a second content playback associated with a second region (e.g., a 25Hz or 50Hz playback rate of EU content), the reference rate 114 may change. A person of ordinary skill in the art will understand that in other specific implementations, the reference rate 114 may change or vary for countless reasons.

[0040] In response to detecting a change in reference rate 114 from the first reference rate to the second reference rate, MSG 120 generates updated synchronization signals 132 and 136 based on the second reference rate and provides updated synchronization signals 132 and 136 to image capture device 142 and display pipeline 146, respectively, to synchronize their clocks based on the second reference rate.

[0041] Figure 1B It is implemented according to some specific Figure 1A 142 and ISP pipeline 144. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, Figure 1B Illustrated Figure 1A The first part of the content generation and delivery architecture 100 includes an image capture device 142 and an ISP pipeline 144.

[0042] like Figure 1BAs shown, the image capture device 142 includes a lens assembly 152 that focuses photons 151 onto a photodiode 154 (e.g., a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor). The photodiode 154 generates a RAW analog image 155 that is fed to a front-end architecture 160, which includes an analog amplifier 162 and an analog-to-digital converter (ADC) 164. A correction engine 166 performs one or more operations and / or algorithms on the output of the ADC 164, such as lens correction 167A, defective pixel correction 167B, etc. As such, as shown in FIG. Figure 1B As shown, the output of the image capture device 142 is a color filter array (CFA) mosaic 168, which may also be referred to in the art as "RAW Bayer input" or "RAW image data."

[0043] like Figure 1B As shown, the ISP pipeline 144 reads the CFA mosaic 168 from the image capture device 142 into a mosaic buffer 169. As one example, the mosaic buffer 169 corresponds to a first-in-first-out (FIFO) buffer, etc. In some implementations, the mosaic buffer 169 corresponds to a single buffer or multiple buffers.

[0044] The ISP pipeline 144 then performs one or more image processing operations and / or algorithms on the CFA mosaic 168, such as white balance 170, debayering / demosaicking 171, noise reduction 172, color correction 173, gamma correction 174, and sharpening 175 to produce RGB data portions 176. In some implementations, the RGB data portions 176 are accumulated in the RGB data buffer 177 until the RGB combiner 178 combines the RGB data portions 176 from the RGB data buffer 177 into an RGB image frame 179 and writes the RGB image frame 179 to the partial frame buffer 145. In some implementations, the RGB data portions 176 bypass the RGB data buffer 177 and are written to the partial frame buffer 145. Figure 2B Describes parts of the frame buffer in more detail.

[0045] In some implementations, the RGB data buffer 177 corresponds to a single buffer or multiple buffers. As described above, those skilled in the art will appreciate that in various implementations, the RGB data portion 176 and the RGB image frame 179 may be replaced with data portions and image frames associated with various other color spaces other than RGB, such as YCbCr, CMYK, etc.

[0046] Those skilled in the art will appreciate that Figure 1B The operations and / or algorithms described are exemplary only, and other operations and / or algorithms may be performed in various other specific implementations. Figure 1B The order of operations and / or algorithms described is exemplary only, and the operations and / or algorithms may be performed in other orders, sequences, and / or in parallel in various other implementations.

[0047] Figure 1C It is implemented according to some specific Figure 1A 146 and a second portion of the content generation and delivery architecture 100 of FIG. 148 including the display pipeline 146 and the display device 148. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, Figure 1C Illustrated Figure 1A The second portion of the content generation and delivery architecture 100 includes a display pipeline 146 and a display device 148 .

[0048] like Figure 1C As shown, the display pipeline 146 includes a renderer 186 that renders virtual content from a virtual content library 180 according to a viewing angle corresponding to a camera pose 182 of the image capture device 142 relative to the physical environment 105 during a current time period and, optionally, past rendering iterations of virtual content from a past rendering buffer 184. The display pipeline 146 also includes a compositor 190 that composites the rendered virtual content with the RGB image frame 179 based on depth information associated with the physical environment 105 from a depth buffer 188 to generate a composite image for a graphics environment (also sometimes referred to herein as an "extended reality (XR) environment") stored in a scan output buffer 192.

[0049] In some implementations, the scan output buffer 192 stores one or more composite images or other content for a graphics environment (sometimes referred to herein as an "XR environment"). In one example, the scan output buffer 192 corresponds to a ping-pong buffer including a front buffer 192A and a back buffer 192B. Those of ordinary skill in the art will appreciate that in various other implementations, the scan output buffer 192 may be structured in different ways. The display device 148 then displays a composite image associated with the state of the graphics environment during the current time period. In some implementations, the display device 148 corresponds to an integrated display. In some implementations, the display device 148 corresponds to an external display. In some implementations, the display device 148 corresponds to a cloud-based recorder, re-encoder, etc. accessible to an end-user device.

[0050] Figure 2A Examples of some specific implementations and Figure 1A 200 associated with the content generation and delivery architecture 100 in FIG. 200. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, Figure 2A Illustrated with Figure 1A A latency graph 200 associated with the content generation and delivery architecture 100 in FIG.

[0051] like Figure 2A As shown, the delay diagram 200 includes a delay diagram for capturing an input image (eg, Figure 1B and an image capture device latency 202 associated with a first amount of time for reading out an input image from the image capture device 142 (e.g., Figure 1B An image capture device readout latency 204 associated with a second amount of time for reading the input image into the CFA mosaic 168 in the ISP pipeline 144.

[0052] exist Figure 2A In the example, the delay graph 200 also includes a delay graph for performing one or more image processing operations (e.g., Figure 1B The latency graph also includes an ISP pipeline latency 205 associated with a third amount of time for the processor to generate a processed image and write the processed image to the portion of the frame buffer 145 using operations 174 to 184 in the flowchart. The latency graph also includes a display pipeline latency 206 associated with a fourth amount of time for reading the processed image from the portion of the frame buffer 145 and generating a composite image based at least in part on the processed image. The latency graph also includes a display device latency 208 associated with a fifth amount of time for presenting the composite image via the display device 148.

[0053] like Figure 2A As shown, the photon-to-photon latency 210 is associated with the amount of time from photon collection (e.g., capture of an input image by the image capture device 142) to photon presentation (e.g., presentation of a composite image by the display device 148). According to some implementations, the photon-to-photon latency 210 takes into account a content production rate (e.g., an image capture rate of the image capture device 142 and an image processing rate of the ISP pipeline 144) and a content consumption rate (e.g., a rendering and composition rate of the display pipeline 146 and a refresh or frame rate of the display device 148).

[0054] According to some implementations, the content generation and delivery architecture 100 determines the time offset 112 in order to reduce the photon-to-photon delay 210 and also takes into account the jitter delay 212 (e.g., buffering time to account for jitter, lag, etc.) and miscellaneous time padding 214. For example, the time offset 112 corresponds to the sum of the photon-to-photon delay 210, the jitter delay 212, and the miscellaneous time padding 214.

[0055] Figure 2B According to some specific implementation Figure 1A 145 associated with the content generation and delivery architecture 100 in FIG. 146 . Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, Figure 1A The content generation and delivery architecture 100 in includes a buffer allocator 260 configured to dynamically allocate memory for a portion of the frame buffer 145 .

[0056] like Figure 2B As shown, the buffer allocator 260 determines the allocation value 262 (e.g., the size of the memory footprint of the portion of the frame buffer 145) based on one or more buffer allocation factors 265 such as: the reference rate 114, the exposure length associated with the image capture device 142, Figure 2A The image capture device readout delay 204, as shown Figure 2A The ISP pipeline latency 205, software jitter, and resolution values ​​associated with the display device 148 are shown by Figure 1A One or more downstream perspective correction operations performed by the content generation and delivery architecture 100 in Figure 1A 148, the content generation and delivery architecture 100 in the buffer allocator 260 may include one or more image processing operations performed by the content generation and delivery architecture 100 in the buffer allocator 260 (e.g., a warping operation that takes into account various factors such as: chromatic aberration from a lens associated with the display device 148, calibration of the image capture device 142, the user's gaze direction, etc.). In addition, in some specific implementations, in response to detecting a change in the reference rate 114 from the first reference rate to the second reference rate, the buffer allocator 260 determines the updated allocation value 262 based at least in part on the second reference rate.

[0057] Figure 3A 3 is a block diagram of an example content generation and delivery architecture 310 with multiple image capture devices according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 3AThe content generation and delivery architecture 310 in is similar to Figure 1A The content generation and delivery architecture 100 in and adapted from it. To this end, Figure 1A and Figure 3A Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0058] like Figure 3A As shown, the content generation and delivery architecture 310 includes a first image capture device 142A and a second image capture device 142B. For example, the first image capture device 142A and the second image capture device 142B correspond to different perspectives of the physical environment 105. Figure 3A In the embodiment of the present invention, MSG 120 provides synchronization signal 132 to first image capture device 142A and second image capture device 142B to synchronize their clocks with display pipeline 146 based on reference rate 114.

[0059] Figure 3B 3 is a block diagram of another example content generation and delivery architecture 320 with multiple image capture devices according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 3B The content generation and delivery architecture 320 in is similar to Figure 1A The content generation and delivery architecture 100 in and adapted from it. To this end, Figure 1A and Figure 3B Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0060] like Figure 3B As shown, the content generation and delivery architecture 310 includes a first image capture device 142A and a second image capture device 142B. For example, the first image capture device 142A and the second image capture device 142B correspond to different perspectives of the physical environment 105.

[0061] exist Figure 3B In the embodiment of the present invention, the MSG 120 generates a synchronization signal 132A based on the time offset 112 and the reference rate 114. The MSG 120 provides the synchronization signal 132A to the first image capture device 142A to synchronize its clock with the display pipeline 146 based on the reference rate 114.

[0062] exist Figure 3BIn the embodiment, MSG 120 also generates a synchronization signal 132B based on the time offset 112 and the reference rate 114. MSG 120 provides the synchronization signal 132B to the second image capture device 142B to synchronize its clock with the display pipeline 146 based on the reference rate 114.

[0063] Figure 3C is a block diagram of an example content generation and delivery architecture 330 with multiple display devices according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 3C The content generation and delivery architecture 330 in is similar to Figure 1A The content generation and delivery architecture 100 in and adapted from it. To this end, Figure 1A and Figure 3C Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0064] like Figure 3C As shown, the content generation and delivery architecture 330 includes a first display device 148A and a second display device 148B. For example, the first display device 148A corresponds to a display for a first eye of a user, and the second display device 148B corresponds to a display for a second eye of the user. In some implementations, the display pipeline 146 can adjust the composite image for stereoscopic presentation via the first display device 148A and the second display device 148B. As such, reference Figure 3C , MSG 120 synchronizes the display of content on physical devices such as display devices 148A and 148B.

[0065] Figure 3D 3 is a block diagram of another example content generation and delivery architecture 340 with multiple display devices according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 3D The content generation and delivery architecture 340 in is similar to Figure 1A The content generation and delivery architecture 100 in and adapted from it. To this end, Figure 1A and Figure 3D Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0066] like Figure 3DAs shown, the content generation and delivery architecture 340 includes a first display device 148A and a second display device 148B. For example, the first display device 148A is associated with a first eye of a user, and the second display device 148B is associated with a second eye of the user. Figure 3D , the content generation and delivery architecture 340 includes a first display pipeline 146A associated with a first display device 148A and a second display pipeline 146B associated with a second display device 148B.

[0067] exist Figure 3D In FIG. 1 , MSG 120 generates synchronization signal 136A based on time offset 112 and reference rate 114. MSG 120 provides synchronization signal 136A to first display pipeline 146A to synchronize its clock with image capture device 142 based on reference rate 114. Figure 3D In the embodiment, MSG 120 also generates a synchronization signal 136B based on the time offset 112 and the reference rate 114. MSG 120 provides the synchronization signal 136B to the second display pipeline 146B to synchronize its clock with the image capture device 142 based on the reference rate 114. Figure 3D , MSG 120 synchronizes content rendering on display pipelines 146A and 146B and synchronizes content display on display devices 148A and 148B.

[0068] Figure 3E 3 is a block diagram of an example content generation and delivery architecture 350 with multiple image capture devices and multiple display devices according to some implementations. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 3E The content generation and delivery architecture 350 in is similar to Figure 1A The content generation and delivery architecture 100 in and adapted from it. To this end, Figure 1A and Figure 3E Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0069] like Figure 3E As shown, the content generation and delivery architecture 350 includes parallel pipelines. Figure 3E In the embodiment, the first pipeline of the content generation and delivery architecture 350 includes a first image capture device 142A, a first ISP pipeline 144A, a first partial frame buffer 145A, a first display pipeline 146A, and a first display device 148A. Figure 3E, the second pipeline of the content generation and delivery architecture 350 includes a second image capture device 142B, a second ISP pipeline 144B, a second partial frame buffer 145B, a second display pipeline 146B, and a second display device 148B.

[0070] For example, first image capture device 142A and second image capture device 142B correspond to different perspectives of physical environment 105. For example, first display device 148A is associated with a first eye of a user, and second display device 148B is associated with a second eye of the user.

[0071] like Figure 3E As shown, the first MSG 120A obtains (eg, receives, retrieves, generates, determines, etc.) a first time offset 112A associated with a first pipeline and a first reference rate 114A associated with the first pipeline, such as a refresh rate or frame rate of a first display device 148A.

[0072] exist Figure 3E , the first MSG 120A generates a synchronization signal 132A based on the first time offset 112A and the first reference rate 114A. The first MSG 120A provides the synchronization signal 132A to the first image capture device 142A to synchronize its clock with the first display pipeline 146A based on the first reference rate 114A.

[0073] exist Figure 3E , the first MSG 120A generates a synchronization signal 136A based on the first time offset 112A and the first reference rate 114A. The first MSG 120A provides the synchronization signal 136A to the first display pipeline 146A to synchronize its clock with the first image capture device 142A based on the first reference rate 114A.

[0074] Similarly, if Figure 3E As shown, the second MSG 120B obtains (eg, receives, retrieves, generates, determines, etc.) a second time offset 112B associated with the second pipeline and a second reference rate 114B associated with the second pipeline, such as a refresh rate or frame rate of a second display device 148B.

[0075] exist Figure 3E In the embodiment, the second MSG 120B generates a synchronization signal 132B based on the second time offset 112B and the second reference rate 114B. The second MSG 120B provides the synchronization signal 132B to the second image capture device 142B to synchronize its clock with the second display pipeline 146B based on the second reference rate 114B.

[0076] exist Figure 3EIn the embodiment, the second MSG 120B generates a synchronization signal 136B based on the second time offset 112B and the second reference rate 114B. The second MSG 120B provides the synchronization signal 136B to the second display pipeline 146B to synchronize its clock with the second image capture device 142B based on the second reference rate 114B.

[0077] In various specific implementations, such as Figure 3E As shown, the content generation and delivery architecture 350 includes an optional cross-pipeline logic 390 for synchronizing the first pipeline and the second pipeline. For example, the cross-pipeline logic 390 generates a cross-pipeline synchronization signal 392 and provides the cross-pipeline synchronization signal 392 to the first MSG 120A and the second MSG 120B. It will be appreciated by those skilled in the art that the content generation and delivery architecture 350 may be modified in numerous ways in various other implementations. As such, reference is made to Figure 3E , MSG 120 synchronizes content rendering and display on physical devices such as first and second pipelines.

[0078] Figure 4A An example display-related interval sequence 410 associated with transitioning from a first frame rate to a second frame rate according to some implementations is illustrated. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as an example, Figure 1A The content generation and delivery architecture 100 or a component thereof (e.g., transition logic 122) detects a transition from a first reference rate (e.g., a 100 Hz display refresh or frame rate) to a second reference rate (e.g., a 90 Hz display refresh or frame rate). Continuing with this example, in response to detecting the transition in the reference rate, Figure 1A The content generation and delivery architecture 100 or components thereof (eg, the transition logic 122 ) causes the display pipeline 146 and / or the display device 148 to operate according to the display-related interval sequence 410 .

[0079] exist Figure 4A , when the display refresh or frame rate corresponds to 100 Hz, the display-related interval 400A includes a first amount of time 402 for a vertical blank display operation (VBLANK), a second amount of time 404 for a vertical active display operation (VACTIVE), and a third amount of time 406A for an idle display operation (IDLE).

[0080] exist Figure 4A, when the display refresh or frame rate corresponds to 90 Hz, the display-related interval 400B includes a first amount of time 402 of a vertical blank period (VBLANK) for a scan-out operation, a second amount of time 404 of a vertical active period (VACTIVE) for the scan-out operation, and a third amount of time 406B of an idle period (IDLE) for the scan-out operation.

[0081] like Figure 4A As shown, display-related intervals 400A and 400B include the same amount of time 402 for VBLANK and the same amount of time 404 for VACTIVE. However, display-related intervals 400A and 400B include different amounts of time 406A and 406B, respectively, for IDLE. Figure 4A , time amount 406B is greater than time amount 406A. In other words, VBLANK and VACTIVE are associated with static values, while IDLE is associated with a dynamic value that is adjusted based on a reference rate (eg, a display refresh or frame rate).

[0082] For example, in response to detecting a transition from a first reference rate (e.g., a 100 Hz display refresh or frame rate) to a second reference rate (e.g., a 90 Hz display refresh or frame rate), Figure 1A The content generation and delivery architecture 100 in the embodiment updates the synchronization signal 136 provided to the display pipeline 146 and / or the display device 148. In response to obtaining the updated synchronization signal 136, the display pipeline 146 and / or the display device 148 reduces its clock frequency from 100 Hz to 90 Hz across the display-related interval sequence 410 by slowly increasing the amount of time associated with IDLE. For example, referring to Figure 4A , Figure 1A The content generation and delivery architecture 100 in the embodiment of the present invention reduces the clock frequency of the display pipeline 146 and / or the display device 148 by approximately 0.9 Hz to 1.0 Hz per interval in the display-related interval sequence 410. As such, in some implementations, the content generation and delivery architecture 100 or components thereof (e.g., Figure 1A The MSG 120 in the embodiment is configured to detect a transition from a first reference rate to a second reference rate (eg, an increase or decrease to the reference rate) and in response thereto dynamically update the synchronization signal to smoothly transition from the first reference rate to the second reference rate.

[0083] Figure 4BAn example time offset between the beginning of the image capture related interval sequence 420 relative to the beginning of the display related interval sequence 410 according to some specific implementations is illustrated. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as an example, Figure 1A The content generation and delivery architecture 100 or a component thereof (e.g., transition logic 122) detects a transition from a first reference rate (e.g., a 100 Hz display refresh or frame rate) to a second reference rate (e.g., a 90 Hz display refresh or frame rate). Continuing with this example, in response to detecting the transition in the reference rate, Figure 1A The content generation and delivery architecture 100 or components thereof (e.g., the transition logic 122) causes the display pipeline 146 and / or display device 148 to operate according to the display-related interval sequence 410 and also causes the image capture device 142 to operate according to the image capture-related interval sequence 420.

[0084] For example, refer to Figure 4B , Figure 1A The content generation and delivery architecture 100 or a component thereof (e.g., the transition logic 122) in the display pipeline 146 and / or the display device 148 reduces the clock frequency by approximately 0.9 Hz to 1.0 Hz per interval in the display-related interval sequence 410. Similarly, with continued reference to Figure 4B , Figure 1A The content generation and delivery architecture 100 or a component thereof (e.g., the transition logic 122) in FIG. 100 reduces the clock frequency of the image capture device 142 by approximately 0.9 Hz to 1.0 Hz per interval in the image capture related interval sequence 420. Figure 4B , the transition from the first reference rate to the second reference rate is completed at time 426 (e.g., when the display-related interval sequence 410 ends). Figure 4B As shown, the start time 424 of the display-related interval sequence 410 is delayed relative to the start time 422 of the image-capture-related interval sequence 420 by a time offset 452 .

[0085] FIG. 5A to FIG. 5C Various timing diagrams 510 , 520 , and 530 are illustrated having different time offsets 515 , 525 , and 535 , respectively, according to some implementations. Figure 5A An example timing diagram 510 with a time offset 515 according to some implementations is illustrated. While certain specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein.

[0086] like Figure 5AAs shown, the ISP write operation 512A for frame N begins at time 0 (e.g., the beginning of the frame N duration 516A), and the display read operation 514A for frame N begins after a time offset 515 relative to the beginning of the ISP write operation 512A for frame N. For example, the ISP write operation 512A for frame N corresponds to writing the processed image associated with frame N to Figure 1A ISP pipeline 144 of the portion of frame buffer 145 in FIG. For example, display read operation 514A of frame N corresponds to reading from Figure 1A The portion of the frame buffer 145 in the display pipeline 146 reads the processed image associated with frame N.

[0087] Similarly, in Figure 5A , the ISP write operation 512B of frame N+1 begins after the end of frame N duration 516A (e.g., the beginning of frame N+1 duration 516B), and the display read operation 514B of frame N+1 begins after a time offset 515 relative to the beginning of the ISP write operation 512B of frame N+1. Figure 5A As shown, Figure 1A The content generation and delivery architecture 100 in the embodiment of the present invention determines the time offset 515 so that the display read operation cannot catch up with the ISP write operation while reducing the time caused by Figure 2B The size of the portion of the frame buffer 145 allocated by the buffer allocator 260 in FIG.

[0088] Figure 5B Another example timing diagram 520 with a time offset 525 according to some implementations is illustrated. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 5B The timing diagram 520 in FIG. 5 is similar to Figure 5A The timing diagram 510 in FIG. 5 is adapted from it. Figure 5A and Figure 5B Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0089] For example, Figure 5B The time offset 525 is less than Figure 5A Thus, the display read operation 514A of frame N ends before the ISP write operation 512A of frame N. In this example, a hardware failure may occur when the display pipeline 146 exhausts the rows written to the portion of the frame buffer 145 by the ISP pipeline 144.

[0090] Figure 5CAnother example timing diagram 530 with a time offset 535 according to some implementations is illustrated. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. According to some implementations, Figure 5C The timing diagram 530 in FIG. 5 is similar to Figure 5A The timing diagram 510 in FIG. 5 is adapted from it. Figure 5A and Figure 5C Similar reference numerals are used throughout, and for the sake of brevity, only the differences between them will be discussed.

[0091] For example, Figure 5C The time offset 535 is greater than Figure 5A The time shift in is 515. Therefore, the photon-to-photon delay is Figure 5A , and the ISP write operation 512A of frame N ends significantly earlier than the display read operation 514A of frame N. In this example, Figure 2B The buffer allocator 260 in may be relative to Figure 5C The timing diagram 530 in FIG. 5 is a diagram showing the allocation of a portion of the frame buffer 145 and the Figure 5A The timing diagram 510 in FIG. 5 shows that the read operation will not be as large as the time offset 535 due to being greater than the time offset 515. Figure 5A Thus, since time offset 535 is greater than time offset 515, compared Figure 5A , photon-to-photon delay and buffer size in Figure 5C In some implementations, the content generation and delivery architecture determines / selects the time offset in order to reduce the photon-to-photon latency and also reduce the size of the partial frame buffer without exhausting the partial frame buffer, as discussed above.

[0092] Figure 6 1 is a flowchart representation of a method 600 for synchronizing a content generation and delivery architecture to reduce latency associated with image pass-through according to some implementations. In various implementations, the method 600 is performed by a computing system comprising: a content generation and delivery architecture including an image capture device, an image signal processing (ISP) pipeline, a portion of a frame buffer, a display pipeline, and a display device; synchronization circuits (e.g., MSG 120 in FIG. 1 ); one or more processors; and a non-volatile memory (e.g., Figure 7In some implementations, method 600 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 600 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). In various implementations, some operations in method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0093] In some implementations, the image capture device includes a single image sensor. In some implementations, the image capture device includes multiple image sensors. In some implementations, the display device includes a single display device. In some implementations, the display device includes multiple display devices.

[0094] As represented by block 602, method 600 includes determining a time offset associated with a content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., MSG 120) determines / selects the time offset 112 in order to reduce the above reference Figure 2A The described photon-to-photon delay 210 also reduces the memory footprint of the portion of the frame buffer 145. Figure 5A , a computing system or a component thereof (e.g., Figure 1A The MSG 120 in FIG. 140 ) determines / selects a time offset 515 in order to reduce the photon-to-photon latency across the content generation and delivery architecture and also reduce the memory footprint of the portion of the frame buffer 145 . Figure 5B , the time offset 525 is too short, which may cause hardware failure when the display pipeline 146 exhausts the lines written by the ISP pipeline 144 to the partial frame buffer 145. Figure 5C , time offset 535 is too long, compared Figure 5A The time offset 515 in the frame buffer 145 may result in a longer photon-to-photon delay and a larger memory footprint for the portion of the frame buffer 145 .

[0095] In some implementations, the time offset also includes jitter delay. In some implementations, the time offset also includes deterministic or non-deterministic time filling. For example, Figure 2A As shown, time offset 112 includes photon-to-photon delay 210 , jitter delay 212 , and miscellaneous time padding 214 .

[0096] As represented by block 604, method 600 includes obtaining a first reference rate associated with a portion of a content generation and delivery architecture. Figure 1A, the content generation and delivery architecture 100 or a component thereof (e.g., the MSG 120) obtains (e.g., receives, retrieves, determines, etc.) a reference rate 114. As an example, the reference rate Figure 1A In a display-centric configuration, the first reference rate corresponds to the current refresh or frame rate of the display device 148. Figure 1A , in the image capture-centric configuration, the first reference rate corresponds to the current shutter speed or image capture rate of the image capture device 142.

[0097] As represented by block 606, method 600 includes generating, via a synchronization circuit, a synchronization signal for a content generation and delivery architecture based at least in part on a first reference rate. Figure 1A The content generation and delivery architecture 100 or a component thereof (eg, the MSG 120 ) obtains (eg, receives, retrieves, determines, generates, etc.) synchronization signals 132 and 136 based at least in part on the reference rate 114 .

[0098] In some implementations, the MSG 120 generates the same synchronization signal that is provided to all parts of the content generation and delivery architecture. For example, the synchronization signal includes a reference rate 114 and a time offset 112. In some implementations, the MSG 120 derives a different synchronization signal for each part of the content generation and delivery architecture based on the reference rate and the adjustable timing-related parameters (e.g., intrinsic parameters) associated with each part of the content generation and delivery architecture. As an example, the adjustable timing-related parameters associated with the image capture device include shutter speed, exposure length, etc. For another example, the adjustable timing-related parameters associated with the display pipeline 146 and / or the display device 148 include the reference rate 114 and the time offset 112 described above. Figure 4A The IDLE period described.

[0099] As represented by block 608, method 600 includes operating the content generation and delivery architecture in accordance with the synchronization signal and the time offset. Figure 1A , the content generation and delivery architecture 100 or a component thereof (eg, the MSG 120) provides synchronization signals 132 and 136 to the image capture device 142 and the display pipeline 146, respectively.

[0100] In some specific implementations, operating the content generation and delivery architecture includes: obtaining an input image of a physical environment from an image capture device based on a synchronization signal; generating a processed image by performing one or more image processing operations on the input image via an ISP pipeline; writing the processed image to a partial frame buffer via the ISP pipeline; reading the processed image from the partial frame buffer via a display pipeline based on the synchronization signal; generating a composite image by synthesizing rendered virtual content with the processed image via the display pipeline; and presenting the composite image or other content via a display device in response to determining that a time offset has passed.

[0101] For example, refer to Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., the image capture device 142) obtains an input image of the physical environment 105 in accordance with the synchronization signal 132. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., the ISP pipeline 144) generates a processed image by performing one or more image processing operations on the input image and writes the processed image to a partial frame buffer. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., the display pipeline 146) reads the processed image from the partial frame buffer according to the synchronization signal 136 and generates a composite image by compositing the rendered virtual content with the processed image. Figure 1A , the content generation and delivery architecture 100 or a component thereof (eg, the display device 148 ) presents the composite image or other content.

[0102] In some implementations, the one or more image processing operations include at least one of a white balance operation, a debayering operation, a noise reduction operation, a color correction operation, a gamma correction operation, and a sharpening operation. Figure 1B A number of image processing operations 174 - 184 associated with the ISP pipeline 144 are illustrated.

[0103] According to some specific implementations, method 600 includes: determining a size value of a portion of a frame buffer based on a reference rate and intrinsic parameters of a display device; and allocating memory for the portion of the frame buffer based on the size value. For example, the intrinsic parameters of the display device include display resolution, aspect ratio, etc. For example, the reference rate and intrinsic parameters of the display device include a display resolution, aspect ratio, etc. Figure 2B , the buffer allocator 260 determines the allocation value 262 (e.g., the size of the memory footprint of the portion of the frame buffer 145) based on: the reference rate 114, the exposure length associated with the image capture device 142, such as Figure 2A The image capture device readout delay 204, as shown Figure 2A The ISP pipeline latency 205, software jitter, and resolution values ​​associated with the display device 148 are shown by Figure 1A One or more downstream perspective correction operations performed by the content generation and delivery architecture 100 in Figure 1A 148, the content generation and delivery architecture 100 in the buffer allocator 260 may include one or more image processing operations performed by the content generation and delivery architecture 100 in the buffer allocator 260 (e.g., a warping operation that takes into account various factors such as: chromatic aberration from a lens associated with the display device 148, calibration of the image capture device 142, the user's gaze direction, etc.). In addition, in some specific implementations, in response to detecting a change in the reference rate 114 from the first reference rate to the second reference rate, the buffer allocator 260 determines the updated allocation value 262 based at least in part on the second reference rate.

[0104] According to some implementations, as represented by block 610, method 600 includes: detecting a change from a first reference rate to a second reference rate; and in response to detecting the change from the first reference rate to the second reference rate, generating an updated synchronization signal for a content generation and delivery architecture based at least in part on the second reference rate. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., the transition logic 122) detects a change in the reference rate 114 from a first reference rate to a second reference rate. As one example, the reference rate 114 increases from 90 Hz to 100 Hz. As another example, the reference rate 114 decreases from 100 Hz to 90 Hz.

[0105] In some implementations, as represented by block 612, a change from a first reference rate to a second reference rate corresponds to a change from a first application to a second application different from the first application. Figure 1A , the content generation and delivery architecture 100 or a component thereof (eg, the transition logic 122 ) detects a transition from a productivity application to a content consumption application or vice versa.

[0106] In some implementations, as represented by block 614, a change from a first reference rate to a second reference rate corresponds to a change from a first user state to a second user state different from the first user state. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., the transition logic 122) detects a transition from a sitting state to a standing state or vice versa based on motion sensor data from at least one of an inertial measurement unit (IMU), accelerometer, gyroscope, magnetometer, etc. of the computing system.

[0107] In some implementations, as represented by block 616, a change from the first reference rate to the second reference rate corresponds to a change from a first environmental state to a second environmental state different from the first environmental state. Figure 1A, the content generation and delivery architecture 100 or a component thereof (e.g., transition logic 122) detects a transition from an indoor environment to an outdoor environment or vice versa based on the location data. Figure 1A , the content generation and delivery architecture 100 or a component thereof (e.g., transition logic 122) detects a transition from a bright environment to a dim environment or vice versa based on an ambient light value from an ambient light sensor of the computing system. Figure 1A , the content generation and delivery architecture 100 or a component thereof (eg, the transition logic 122 ) detects a transition from a high frequency environment to a low frequency environment or vice versa based on characterization data or image processing data associated with the physical environment 105 .

[0108] Figure 7 is a block diagram of an example of a computing system 700 according to some implementations. While certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, by way of non-limiting example, in some implementations, the computing system 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more display devices 146, one or more image capture devices 142, one or more depth sensors 716, memory 720, and one or more communication buses 704 for interconnecting these and various other components. In various implementations, the computing system 700 includes reference Figure 1A The content generation and delivery architecture 100 described herein. In some implementations, the computing system 700 provides Figure 1A Functionality of the content generation and delivery architecture 100 in.

[0109] In some implementations, the one or more communication buses 704 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 706 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, an ambient light sensor, one or more environmental sensors, etc.

[0110] In some implementations, one or more display devices 146 are configured to provide a user interface or XR experience to a user. In some implementations, one or more display devices 146 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more display devices 146 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays.

[0111] In some implementations, the one or more image capture devices 142 include: Figure 1B The lens assembly 152, photodiode 154, and front-end architecture 160 are shown. The one or more image capture devices 142 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor CMOS image sensors or CCD image sensors), one or more infrared (IR) cameras, one or more event-based cameras, etc. In some specific implementations, the one or more depth sensors 716 correspond to structured light devices, time of flight devices, etc.

[0112] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules and data structures or a subset thereof.

[0113] Operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks.

[0114] In some implementations, the data obtainer 732 is configured to obtain data (e.g., image data, presentation data, interaction data, sensor data, position data, etc.) from at least one or more I / O devices and sensors 706, one or more image capture devices 142, one or more depth sensors 716, etc. To this end, in various implementations, the data obtainer 732 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0115] In some implementations, the data transmitter 734 is configured to transmit data (e.g., image data, sensor data, presentation data, location data, etc.) to a local or remote recipient. To this end, in various implementations, the data transmitter 734 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0116] In some implementations, the MSG 120 is configured to determine / select the time offset 112 so as to reduce the above reference Figure 2A The described photon-to-photon delay 210 and also reduce the memory footprint of the portion of the frame buffer 145. In some implementations, the MSG 120 is further configured to generate a synchronization signal based on the reference rate and the time offset. Figure 1A The MSG 120 is described in more detail. In some implementations, the MSG 120 includes transition logic 122 for handling changes in the reference rate. To this end, in various implementations, the MSG 120 includes instructions and / or logic for such instructions as well as heuristics and metadata for the heuristics.

[0117] In some implementations, the ISP pipeline 144 is configured to obtain an input image from one or more image capture devices 142 and generate a processed image by performing one or more image processing operations on the input image. In some implementations, the ISP pipeline 144 is also configured to write the processed image to the partial frame buffer 145. Figure 1B The ISP pipeline 144 is described in greater detail. To this end, in various implementations, the ISP pipeline 144 includes instructions and / or logic for such instructions as well as heuristics and metadata for the heuristics.

[0118] In some implementations, the buffer allocator 260 is configured to determine a size value of the portion of the frame buffer 145 and allocate memory for the portion of the frame buffer 145 based on the size value. Figure 2BThe buffer allocator 260 is described in greater detail. To this end, in various implementations, the buffer allocator 260 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0119] In some implementations, the display pipeline 146 is configured to read the processed image from the portion of the frame buffer. In some implementations, the display pipeline 146 is further configured to generate a composite image by compositing the processed image frame with the rendered virtual content. Figure 1C The display pipeline 146 is described in greater detail. To this end, in various implementations, the display pipeline 146 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0120] In some implementations, computing system 700 corresponds to a wearable computing device, a wearable computing device, a head-mounted system, etc. Figure 7 It is intended more as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0121] Fig. 8A 1 is a flowchart representation of a method 800 for synchronizing content generation and delivery architectures to reduce latency associated with image pass-through according to some implementations. In various implementations, the method 800 is performed by an electronic device comprising: an image capture device; a display device; a communication interface for communicating with a controller including synchronization circuits (e.g., MSG 120 in FIG. 1 ), an image signal processing (ISP) pipeline, and a display pipeline; one or more processors; and a non-volatile memory (e.g., Fig. 9 In some implementations, the method 800 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in the method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0122] In some implementations, the image capture device includes a single image sensor. In some implementations, the image capture device includes multiple image sensors. In some implementations, the display device includes a single display device. In some implementations, the display device includes multiple display devices.

[0123] As represented by block 802, method 800 includes transmitting a reference rate associated with a display device to a server via a communication interface. Fig. 9 , the electronic device 900 transmits the reference rate to the controller 950 via one or more communication interfaces 908.

[0124] As represented by block 804, method 800 includes obtaining a time offset to reduce a photon-to-photon delay associated with the amount of time between an image capture device capturing an image and a display device presenting the image. In some implementations, the time offset also reduces a memory footprint of a portion of frame buffer 145. For example, referring to Fig. 9 , the electronic device 900 obtains the time offset via the one or more communication interfaces 908, for example, from the controller 950.

[0125] In some implementations, the time offset also includes jitter delay. In some implementations, the time offset includes deterministic or non-deterministic time padding. For example, Figure 2A As shown, time offset 112 includes photon-to-photon delay 210 , jitter delay 212 , and miscellaneous time padding 214 .

[0126] As represented by block 806, method 800 includes obtaining a synchronization signal associated with a reference rate from the MSG via the communication interface for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline. Fig. 9 , the electronic device 900 obtains a synchronization signal from the controller 950 via one or more communication interfaces 908 .

[0127] As represented by block 808, method 800 includes obtaining an input image of the physical environment via an image capture device in accordance with the synchronization signal. Fig. 9 , the electronic device 900 obtains an input image from one or more image capture devices 142 .

[0128] As represented by block 810, method 800 includes transmitting an input image to an ISP pipeline via a communication interface. Fig. 9 , the electronic device 900 transmits the input image to the controller 950 via one or more communication interfaces 908 .

[0129] As represented by block 812, method 800 includes obtaining a composite image from a display pipeline via a communication interface, wherein the composite image includes rendered virtual content composited with a representation of an input image. Fig. 9 , the electronic device 900 obtains the processed image from the controller 950 via one or more communication interfaces 908 .

[0130] As represented by block 814, in response to determining that the time offset has elapsed, method 800 includes presenting the composite image via a display device. Fig. 9 , one or more display devices 146 of the electronic device 900 present the composite image after the time offset has passed.

[0131] According to some implementations, method 800 includes: detecting a change from a first reference rate to a second reference rate; in response to detecting the change from the first reference rate to the second reference rate, transmitting the second reference rate to the controller via a communication interface; and obtaining an updated synchronization signal from the MSG via the communication interface based at least in part on the second reference rate. For example, the reference rate Fig. 9 , the electronic device 900 detects a change in the reference rate 114 from the first reference rate to the second reference rate and transmits the second reference rate to the controller 950 via the one or more communication interfaces 908. Continuing with this example, the electronic device 900 obtains an updated synchronization signal from the controller 950 via the one or more communication interfaces 908. As an example, the reference rate 114 increases from 90 Hz to 100 Hz. As another example, the reference rate 114 decreases from 100 Hz to 90 Hz.

[0132] In some implementations, a change from a first reference rate to a second reference rate corresponds to a change from a first application to a second application that is different from the first application. Fig. 9 , the electronic device 900 detects a transition from a productivity application to a content consumption application or vice versa.

[0133] In some implementations, a change from a first reference rate to a second reference rate corresponds to a change from a first user state to a second user state different from the first user state. Fig. 9 , the electronic device 900 detects a transition from sitting to standing or vice versa based on motion data from one or more I / O devices and sensors 906.

[0134] In some implementations, a change from a first reference rate to a second reference rate corresponds to a change from a first environmental state to a second environmental state different from the first environmental state. Fig. 9 , the electronic device 900 detects a transition from an indoor environment to an outdoor environment or vice versa. Fig. 9, the electronic device 900 detects a transition from a bright environment to a dim environment or vice versa based on ambient light values ​​from one or more I / O devices and sensors 906. Fig. 9 , the electronic device 900 detects a transition from a high frequency environment to a low frequency environment or vice versa based on frequency values ​​associated with input images of the physical environment 105 from the one or more image capture devices 142 .

[0135] Figure 8B and Figure 8C A flowchart representation of a method 850 for synchronizing content generation and delivery architecture to reduce latency associated with image pass-through according to some implementations is illustrated. In various implementations, the method 850 is performed by a controller that includes: a synchronization circuit (e.g., MSG 120 in FIG. 1 ); a portion of a frame buffer; an image signal processing (ISP) pipeline; a display pipeline; a communication interface for communicating with an electronic device including an image capture device and a display device; one or more processors; and a non-volatile memory (e.g., Fig. 9 Controller 950 in ). In some specific implementations, method 850 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 850 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). In various specific implementations, some operations in method 850 are optionally combined, and / or the order of some operations is optionally changed.

[0136] As represented by block 852, method 850 includes obtaining a reference rate associated with a display device from an electronic device via a communication interface. Fig. 9 , the controller 950 obtains the reference rate from the electronic device 900 via one or more communication interfaces 958 .

[0137] As represented by block 854, method 850 includes determining a time offset to reduce a photon-to-photon delay associated with the amount of time between the image capture device capturing the image and the display device presenting the image. According to some implementations, MSG 120 determines / selects time offset 112 to reduce the above referenced Figure 2A The described photon-to-photon delay 210 also reduces the memory footprint of the portion of the frame buffer 145. For example, referring to Fig. 9 , controller 950 determines / selects the time offset.

[0138] As represented by block 856, method 850 includes determining a synchronization signal based on the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline. Fig. 9, the controller 950 or a component thereof (eg, MSG 120) determines a synchronization signal.

[0139] As represented by block 858, method 850 includes transmitting the synchronization signal to the electronic device via the communication interface. Fig. 9 , the controller 950 transmits the synchronization signal to the electronic device 900 via one or more communication interfaces 958 .

[0140] As represented by block 860, method 850 includes obtaining an input image of the physical environment from an image capture device via a communication interface in accordance with a synchronization signal. Fig. 9 , the controller 950 obtains an input image from the electronic device 900 via one or more communication interfaces 958 .

[0141] As represented by block 862, method 850 includes generating a processed image by performing one or more image processing operations on an input image via the ISP pipeline. Fig. 9 , the controller 950 or a component thereof (e.g., the ISP pipeline 144) generates a processed image by performing one or more image processing operations on the input image. In some specific implementations, the one or more image processing operations include at least one of a white balance operation, a debayer operation, a noise reduction operation, a color correction operation, a gamma correction operation, and a sharpening operation. For example, Figure 1B A number of image processing operations 174 - 184 associated with the ISP pipeline 144 are illustrated.

[0142] As represented by block 864, method 850 includes writing the processed image to a partial frame buffer via the ISP pipeline. Fig. 9 , the controller 950 or a component thereof (e.g., the ISP pipeline 144) writes the processed image to a partial frame buffer 145.

[0143] As represented by block 866, method 850 includes reading the processed image from the portion of the frame buffer via the display pipeline in accordance with the synchronization signal. Fig. 9 , the controller 950 or a component thereof (eg, the display pipeline 146 ) reads the processed image from the portion of the frame buffer 145 according to the synchronization signal.

[0144] As represented by block 868, method 850 includes generating a composite image via the display pipeline by compositing the rendered virtual content with the processed image. Fig. 9 , the controller 950 or a component thereof (eg, the display pipeline 146) generates a composite image by combining the rendered virtual content with the processed image. Figure 1C The display pipeline 146 is described in more detail.

[0145] As represented by block 866, method 850 includes transmitting the composite image to a display device via the communication interface. Fig. 9 , the controller 950 transmits the composite image to the electronic device 900 via one or more communication interfaces 958 .

[0146] Fig. 9 is a block diagram of an example of an electronic device 900 and a controller 950 according to some specific implementations. In some specific implementations, the electronic device 900 and the controller 950, alone or in combination, provide Figure 1A Although certain specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the electronic device 900 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more display devices 146, one or more image capture devices 142, one or more depth sensors 916, a memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0147] In some implementations, the one or more communication buses 904 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 906 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, an ambient light sensor, one or more environmental sensors, etc.

[0148] In some implementations, one or more display devices 146 are configured to provide a user interface or XR experience to a user. In some implementations, one or more display devices 146 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more display devices 146 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays.

[0149] In some implementations, the one or more image capture devices 142 include: Figure 1B The lens assembly 152, photodiode 154, and front-end architecture 160 are shown. The one or more image capture devices 142 may include one or more RGB cameras (e.g., with a CMOS image sensor or a CCD image sensor), one or more infrared (IR) cameras, one or more event-based cameras, etc. In some implementations, the one or more depth sensors 916 correspond to a structured light device, a time-of-flight device, etc.

[0150] The memory 920 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 920 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 920 or the non-transitory computer-readable storage medium of the memory 920 stores the following programs, modules and data structures or a subset thereof.

[0151] The operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks.

[0152] In some implementations, the data obtainer 932 is configured to obtain data (e.g., synchronization signals, image data, presentation data, interaction data, sensor data, position data, etc.) from at least one or more I / O devices and sensors 906, one or more image capture devices 142, one or more depth sensors 916, controller 950 via one or more communication interfaces 908, etc. To this end, in various implementations, the data obtainer 932 includes instructions and / or logic for these instructions as well as heuristics and metadata for the heuristics.

[0153] In some implementations, the data transmitter 934 is configured to transmit data (e.g., image data, sensor data, presentation data, position data, etc.) to the controller 950 via the one or more communication interfaces 908. To this end, in various implementations, the data transmitter 934 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0154] like Fig. 9 As shown, the electronic device 900 and the controller 950 are communicatively coupled via a wireless or wired communication channel 940. In some implementations, the electronic device 900 corresponds to a wearable computing device, a head-mounted display, a head-mounted system, a tablet computer, a smart phone, etc. In some implementations, the controller 950 corresponds to a local or remote server, a cloud server, a laptop computer, a tablet computer, a smart phone, a wearable computing device, etc.

[0155] exist Fig. 9 , the controller 950 includes one or more processing units 952 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 956, one or more communication interfaces 958 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 960, a memory 970, and one or more communication buses 954 for interconnecting these and various other components.

[0156] In some implementations, one or more communication buses 954 include circuits that interconnect system components and control communications between system components. One or more I / O devices and sensors 956 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, etc.

[0157] The memory 970 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 970 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 970 optionally includes one or more storage devices remotely located from one or more processing units 952. The memory 970 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 970 or the non-transitory computer-readable storage medium of the memory 970 stores the following programs, modules and data structures or a subset thereof.

[0158] The operating system 980 includes procedures for handling various basic system services and for performing hardware-related tasks.

[0159] In some implementations, the data obtainer 982 is configured to obtain data (e.g., image data, presentation data, interaction data, sensor data, location data, etc.) from at least one or more I / O devices and sensors 956, one or more image capture devices 142, one or more depth sensors 916, the electronic device 900 via one or more communication interfaces 958, etc. To this end, in various implementations, the data obtainer 982 includes instructions and / or logic for these instructions as well as heuristics and metadata for the heuristics.

[0160] In some implementations, the data transmitter 984 is configured to transmit data (e.g., synchronization signals, image data, sensor data, presentation data, location data, etc.) to the electronic device 900 via one or more communication interfaces 958. To this end, in various implementations, the data transmitter 984 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0161] In some implementations, the MSG 120 is configured to determine / select the time offset 112 so as to reduce the above reference Figure 2A The described photon-to-photon delay 210 and also reduce the memory footprint of the portion of the frame buffer 145. In some implementations, the MSG 120 is further configured to generate a synchronization signal based on the reference rate and the time offset. Figure 1A The MSG 120 is described in more detail. In some implementations, the MSG 120 includes transition logic 122 for handling changes in the reference rate. To this end, in various implementations, the MSG 120 includes instructions and / or logic for such instructions as well as heuristics and metadata for the heuristics.

[0162] In some implementations, the ISP pipeline 144 is configured to obtain an input image from one or more image capture devices 142 and generate a processed image by performing one or more image processing operations on the input image. In some implementations, the ISP pipeline 144 is also configured to write the processed image to the partial frame buffer 145. Figure 1B The ISP pipeline 144 is described in greater detail. To this end, in various implementations, the ISP pipeline 144 includes instructions and / or logic for such instructions as well as heuristics and metadata for the heuristics.

[0163] In some implementations, the buffer allocator 260 is configured to determine a size value of the portion of the frame buffer 145 and allocate memory for the portion of the frame buffer 145 based on the size value. Figure 2B The buffer allocator 260 is described in greater detail. To this end, in various implementations, the buffer allocator 260 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0164] In some implementations, the display pipeline 146 is configured to read the processed image from the portion of the frame buffer. In some implementations, the display pipeline 146 is further configured to generate a composite image by compositing the processed image frame with the rendered virtual content. Figure 1C The display pipeline 146 is described in greater detail. To this end, in various implementations, the display pipeline 146 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0165] also, Fig. 9 It is intended more as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig. 9 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0166] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, a device and / or a method can be implemented using any number of aspects set forth herein. In addition, in addition to or different from one or more aspects set forth herein, such a device and / or such a method can be implemented using other structures and / or functions.

[0167] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0168] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0169] As used herein, the term "if" may be interpreted to mean "when the antecedent is true" or "when the antecedent is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" is interpreted to mean "upon determining that the antecedent is true" or "in response to determining" or "upon determining" that the antecedent is true or "when detecting that the antecedent is true" or "in response to detecting" that the antecedent is true, depending on the context.

Claims

1. A computing system, comprising: A content generation and delivery architecture comprising an image capture device, an image signal processing (ISP) pipeline, a portion of a frame buffer, a display pipeline, and a display device; Synchronous circuits; one or more processors; non-transitory memory; as well as one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the computing system to: determining a time offset associated with the content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture; obtaining a first reference rate associated with a portion of the content generation and delivery architecture; generating, via the synchronization circuit, a synchronization signal for the content generation and delivery architecture based at least in part on the first reference rate; as well as The content generation and delivery architecture is operated in accordance with the synchronization signal and the time offset.

2. The computing system of claim 1 , wherein operating the content generation and delivery architecture comprises: obtaining an input image of a physical environment from the image capture device in accordance with the synchronization signal; generating a processed image by performing one or more image processing operations on the input image via the ISP pipeline; writing the processed image to the portion of the frame buffer via the ISP pipeline according to the synchronization signal; reading the processed image from the portion of the frame buffer via the display pipeline according to the synchronization signal; generating a composite image by composite the rendered virtual content with the processed image via the display pipeline; as well as In response to determining that the time offset has elapsed, presenting the composite image via the display device in accordance with the synchronization signal.

3. The computing system of claim 2, wherein the one or more image processing operations include at least one of a white balance operation, a debayer operation, a noise reduction operation, a color correction operation, a gamma correction operation, and a sharpening operation.

4. The computing system of any one of claims 1 to 3, wherein the one or more programs further cause the computing system to: Determining a size value of the portion of the frame buffer based on the reference rate and intrinsic parameters of the display device; and Memory is allocated for the portion of the frame buffer based on the size value.

5. The computing system of any one of claims 1 to 4, wherein the one or more programs further cause the computing system to: detecting a change from the first reference rate to a second reference rate; and In response to detecting the change from the first reference rate to the second reference rate, an updated synchronization signal is generated for the content generation and delivery architecture based at least in part on the second reference rate. 6 . The computing system of claim 5 , wherein the change from the first reference rate to the second reference rate corresponds to a change from a first application to a second application different from the first application. 7 . The computing system of claim 5 , wherein the change from the first reference rate to the second reference rate corresponds to a change from a first user state to a second user state different from the first user state.

8. The computing system of claim 5, wherein the change from the first reference rate to the second reference rate corresponds to a change from a first environmental state to a second environmental state different from the first environmental state.

9. A computing system according to any one of claims 1 to 8, wherein generating the synchronization signal for the content generation and delivery architecture includes generating the same synchronization signal for all parts of the content generation and delivery architecture based at least in part on the first reference rate.

10. A computing system according to any one of claims 1 to 8, wherein generating the synchronization signal for the content generation and delivery architecture includes generating a different synchronization signal for each part of the content generation and delivery architecture based at least in part on the first reference rate and adjustable timing-related parameters associated with each part of the content generation and delivery architecture.

11. The computing system of any one of claims 1 to 10, wherein the time offset comprises a jitter delay.

12. The computing system of any one of claims 1 to 11, wherein the time offset comprises a deterministic or non-deterministic time fill.

13. The computing system of any one of claims 1 to 12, wherein the image capture device comprises a plurality of image sensors.

14. The computing system of any one of claims 1 to 13, wherein the display device comprises a plurality of display devices.

15. A method comprising: At a computing system including non-transitory memory and one or more processors, synchronization circuitry, and a content generation and delivery architecture including an image capture device, an image signal processing (ISP) pipeline, a portion of a frame buffer, a display pipeline, and a display device: determining a time offset associated with the content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture; obtaining a first reference rate associated with a portion of the content generation and delivery architecture; generating, via the synchronization circuit, a synchronization signal for the content generation and delivery architecture based at least in part on the first reference rate; as well as The content generation and delivery architecture is operated in accordance with the synchronization signal and the time offset.

16. A non-transitory memory storing one or more programs that, when executed by one or more processors of a computing system having synchronization circuits and a content generation and delivery architecture including an image capture device, an image signal processing (ISP) pipeline, a portion of a frame buffer, a display pipeline, and a display device, causes the computing system to: determining a time offset associated with the content generation and delivery architecture to reduce photon-to-photon latency across the content generation and delivery architecture; obtaining a first reference rate associated with a portion of the content generation and delivery architecture; generating, via the synchronization circuit, a synchronization signal for the content generation and delivery architecture based at least in part on the first reference rate; as well as The content generation and delivery architecture is operated in accordance with the synchronization signal and the time offset.

17. An electronic device, comprising: Image capture device; Display devices; a communication interface for communicating with a controller including a synchronization circuit, an image signal processing (ISP) pipeline, and a display pipeline; one or more processors; non-transitory memory; as well as One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the electronic device to: transmitting a reference rate associated with the display device to the synchronization circuit via the communication interface; obtaining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; obtaining, from the synchronization circuit via the communication interface, a synchronization signal associated with the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; obtaining, via the image capture device, an input image of a physical environment in accordance with the synchronization signal; transmitting the input image to the ISP pipeline via the communication interface; obtaining, from the display pipeline via the communication interface, a composite image, wherein the composite image comprises rendered virtual content composited with a representation of the input image; as well as In response to determining that the time offset has elapsed, presenting the composite image via the display device.

18. The electronic device of claim 17, wherein the one or more programs further cause the electronic device to: detecting a change from the first reference rate to a second reference rate; in response to detecting the change from the first reference rate to the second reference rate, transmitting the second reference rate to the controller via the communication interface; and An updated synchronization signal is obtained from the synchronization circuit via the communication interface based at least in part on the second reference rate. 19 . The electronic device of claim 18 , wherein the change from the first reference rate to the second reference rate corresponds to a change from a first application to a second application different from the first application.

20. The electronic device of claim 18, wherein the change from the first reference rate to the second reference rate corresponds to a change from a first user state to a second user state different from the first user state.

21. The electronic device of claim 18, wherein the change from the first reference rate to the second reference rate corresponds to a change from a first environmental state to a second environmental state different from the first environmental state.

22. The electronic device of any one of claims 17 to 21, wherein the time offset comprises a jitter delay.

23. The electronic device of any one of claims 17 to 22, wherein the time offset comprises a deterministic or non-deterministic time fill.

24. An electronic device as claimed in any one of claims 17 to 23, wherein the image capture device comprises a plurality of image sensors.

25. The electronic device according to any one of claims 17 to 24, wherein the display device comprises a plurality of display devices.

26. A method comprising: At an electronic device including a non-transitory memory and one or more processors, an image capture device, a display device, and a communication interface for communicating with a controller including a synchronization circuit, an image signal processing (ISP) pipeline, and a display pipeline: transmitting a reference rate associated with the display device to the synchronization circuit via the communication interface; obtaining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; obtaining, from the synchronization circuit via the communication interface, a synchronization signal associated with the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; obtaining, via the image capture device, an input image of a physical environment in accordance with the synchronization signal; transmitting the input image to the ISP pipeline via the communication interface; obtaining, from the display pipeline via the communication interface, a composite image, wherein the composite image comprises rendered virtual content composited with a representation of the input image; as well as In response to determining that the time offset has elapsed, presenting the composite image via the display device.

27. A non-transitory memory storing one or more programs that, when executed by one or more processors of an electronic device having an image capture device, a display device, and a communication interface for communicating with a controller including synchronization circuits, an image signal processing (ISP) pipeline, and a display pipeline, cause the electronic device to: transmitting a reference rate associated with the display device to the synchronization circuit via the communication interface; obtaining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; obtaining, from the synchronization circuit via the communication interface, a synchronization signal associated with the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; obtaining, via the image capture device, an input image of a physical environment in accordance with the synchronization signal; transmitting the input image to the ISP pipeline via the communication interface; obtaining, from the display pipeline via the communication interface, a composite image, wherein the composite image comprises rendered virtual content composited with a representation of the input image; as well as In response to determining that the time offset has elapsed, presenting the composite image via the display device.

28. A controller, comprising: Synchronous circuits; Partial frame buffer; Image signal processing (ISP) pipeline; Display pipeline; a communication interface for communicating with an electronic device including an image capture device and a display device; one or more processors; non-transitory memory; as well as one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the controller to: obtaining, from the electronic device via the communication interface, a reference rate associated with the display device; determining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; determining a synchronization signal based on the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; transmitting the synchronization signal to the electronic device via the communication interface; obtaining an input image of a physical environment from the image capture device via the communication interface in accordance with the synchronization signal; generating a processed image by performing one or more image processing operations on the input image via the ISP pipeline; Writing the processed image to the portion of the frame buffer via the ISP pipeline; reading the processed image from the portion of the frame buffer via the display pipeline according to the synchronization signal; generating a composite image by composite the rendered virtual content with the processed image via the display pipeline; as well as The composite image is transmitted to the display device via the communication interface.

29. The controller of claim 28, wherein the one or more programs further cause the controller to: Determining a size value of the portion of the frame buffer based on the reference rate and intrinsic parameters of the display device; and Memory is allocated for the portion of the frame buffer based on the size value.

30. The controller of any one of claims 28 to 29, wherein the one or more programs further cause the controller to: obtaining a second reference rate from the electronic device via the communication interface; and In response to obtaining the change from the second reference rate: generating an updated synchronization signal based at least in part on the second reference rate; and The updated synchronization signal is transmitted to the electronic device via the communication interface.

31. A controller according to any one of claims 28 to 30, wherein the one or more image processing operations include at least one of a white balance operation, a debayer operation, a noise reduction operation, a color correction operation, a gamma correction operation, and a sharpening operation.

32. A controller according to any one of claims 28 to 31, wherein the time offset comprises a jitter delay.

33. A controller according to any one of claims 28 to 32, wherein the time offset comprises a deterministic or non-deterministic time fill.

34. A method comprising: At a controller including non-transitory memory and one or more processors, synchronization circuits, a portion of a frame buffer, an image signal processing (ISP) pipeline, a display pipeline, and a communication interface for communicating with an electronic device including an image capture device and a display device: obtaining, from the electronic device via the communication interface, a reference rate associated with the display device; determining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; determining a synchronization signal based on the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; transmitting the synchronization signal to the electronic device via the communication interface; obtaining an input image of a physical environment from the image capture device via the communication interface in accordance with the synchronization signal; generating a processed image by performing one or more image processing operations on the input image via the ISP pipeline; writing the processed image to the portion of the frame buffer via the ISP pipeline according to the synchronization signal; reading the processed image from the portion of the frame buffer via the display pipeline according to the synchronization signal; generating a composite image by composite the rendered virtual content with the processed image via the display pipeline; as well as The composite image is transmitted to the display device via the communication interface.

35. A non-transitory memory storing one or more programs that, when executed by one or more processors of a controller having synchronization circuits, a portion of a frame buffer, an image signal processing (ISP) pipeline, a display pipeline, and a communication interface for communicating with an electronic device including an image capture device and a display device, cause the controller to: obtaining, from the electronic device via the communication interface, a reference rate associated with the display device; determining a time offset to reduce a photon-to-photon delay associated with an amount of time between the image being captured by the image capture device and the image being presented by the display device; determining a synchronization signal based on the reference rate for synchronizing the image capture device and the display device with the ISP pipeline and the display pipeline; transmitting the synchronization signal to the electronic device via the communication interface; obtaining an input image of a physical environment from the image capture device via the communication interface in accordance with the synchronization signal; generating a processed image by performing one or more image processing operations on the input image via the ISP pipeline; Writing the processed image to the portion of the frame buffer via the ISP pipeline; reading the processed image from the portion of the frame buffer via the display pipeline according to the synchronization signal; generating a composite image by composite the rendered virtual content with the processed image via the display pipeline; as well as The composite image is transmitted to the display device via the communication interface.

Images