Techniques for displaying and capturing images

By using multiple cameras to capture image streams in an extended reality system and applying perspective correction and image correction operations, the image inaccuracy problem caused by camera-user viewpoint differences is solved, and the accuracy of user experience and image display is improved.

CN119071651AActive Publication Date: 2024-12-03APPLE INC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410678001.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-10
Filing Date
2024-05-29
Publication Date
2024-12-03
Estimated Expiration
2044-05-29

AI Technical Summary

Technical Problem

In an extended reality system, when image capture in the user environment, the displayed image is inaccurate due to the uncorrected difference between the camera and the user's viewpoint, which affects the user experience.

Method used

A set of cameras is used to capture the image stream and generate a volume output image collection by applying transformation operations such as perspective correction and image correction to ensure image accuracy and user experience.

Benefits of technology

By generating an accurate stereoscopic output image collection, it reduces user discomfort when viewing images, and improves the user experience of the extended reality system and the accuracy of image display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071651A_ABST
    Figure CN119071651A_ABST
Patent Text Reader

Abstract

Embodiments disclosed herein relate to devices, systems, and methods for individually processing image streams to generate a first set of images for display by a set of displays and a second set of images stored or transmitted as part of a media capture event. Specifically, a first set of images from an image stream may be used to generate a first set of transformed images, and the transformed images may be displayed on a set of displays. A second set of images may be selected from the image stream in response to a capture request associated with a media capture event, and the second set of images may be used to generate a second set of transformed images. A set of output images may be generated from the second set of transformed images, and the set of output images may be stored or transmitted for later viewing.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Provisional Patent Application No. 63 / 470,081 filed on May 31, 2023, and claims the benefit of the U.S. Provisional Patent Application under 35 U.S.C. 119(e), the contents of which are incorporated herein by reference as if fully disclosed herein. Technical Field

[0003] The described embodiments are generally directed to separately processing image streams for real-time display and media capture events. Background Art

[0004] The extended reality system can be used to generate a partially or fully simulated environment (e.g., a virtual reality environment, a mixed reality environment, etc.), where virtual content can replace or enhance the physical world. The simulated environment can provide an engaging experience for the user and is used in games, personal communications, virtual travel, health care, and many other scenarios. In some cases, the simulated environment may include information captured from the user's environment. For example, a pass-through mode of the extended reality system may use one or more displays of the extended reality system to display an image of the user's physical environment. This may allow the user to perceive their physical environment via the display of the extended reality system.

[0005] Images of the user's environment may be captured using one or more cameras of the extended reality system. Depending on the positioning of the cameras in the extended reality system, differences in viewpoint between the user and the cameras, if not corrected, may result in images displayed in pass-through mode that do not accurately reflect the user's physical environment. This may cause user discomfort or may negatively impact the user experience during pass-through mode. Summary of the invention

[0006] Embodiments described herein relate to systems, devices, and methods for performing separate image processing. Some embodiments relate to a method comprising: capturing an image stream using a set of cameras of a device; generating a first set of transformed images from the image stream by applying a first transformation operation to images in the image stream; and displaying the first set of transformed images at a set of displays of the device. The method also includes: receiving a capture request; and generating a set of stereo output images in response to receiving the capture request. Generating the set of stereo output images comprises: selecting a set of images from the image stream; generating a second set of transformed images from the set of images by applying a second transformation operation different from the first transformation operation to the set of images; and generating a set of stereo output images using the second set of transformed images.

[0007] In some variations of these methods, generating the set of stereo output images includes generating a fused stereo output image from two or more image pairs in the set of images. Additionally or alternatively, the set of stereo output images includes a stereo video. In some of these variations, generating the set of stereo output images includes performing a video stabilization operation on the second set of transformed images.

[0008] In some cases, the first transformation operation is a perspective correction operation. Additionally or alternatively, the second transformation operation may be an image correction operation. Generating a set of stereoscopic output images may include generating metadata associated with the set of stereoscopic output images. The metadata may include field of view information for a set of cameras, pose information for the set of stereoscopic output images, and / or a set of default disparity values ​​for the set of stereoscopic output images. In some cases, generating metadata associated with the set of stereoscopic output images includes selecting a set of default disparity values ​​based on a scene captured by the set of stereoscopic output images. In some variants, virtual content is added to the first set of transformed images.

[0009] Some embodiments relate to a device comprising a set of cameras, a set of displays, a memory, and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to perform any of the above methods. Similarly, other embodiments relate to a non-transitory computer-readable medium comprising instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any of the above methods.

[0010] Other embodiments relate to a method that includes capturing a first set of image pairs of a scene using a first camera and a second camera of a device. Each image pair in the first set of image pairs includes a first image captured by the first camera and a second image captured by the second camera. The method includes generating a first set of transformed image pairs from the first set of image pairs, wherein generating the first set of transformed image pairs includes: for each image pair in the first set of image pairs, generating a first perspective-corrected image from the first image based on a difference between a viewpoint of the first camera and a viewpoint of a user; and generating a second perspective-corrected image from the first image based on a difference between a viewpoint of the second camera and a viewpoint of the user. The method also includes displaying the first set of transformed image pairs on a set of displays of the device.

[0011] In addition, the method includes: receiving a capture request; in response to receiving the capture request, selecting a second set of image pairs from the first set of image pairs; and generating a second set of transformed image pairs from the second set of image pairs. This includes, for each image pair in the first set of image pairs: generating a first dewarped image from the first image; generating a second dewarped image from the second first image; and aligning the first dewarped image and the second dewarped image. The method includes generating a set of output images using the second set of transformed image pairs.

[0012] In some variations of these methods, generating the set of output images includes generating a fused stereo output image from two or more image pairs in the set of transformed image pairs. Additionally or alternatively, the set of stereo output images includes a stereo video formed from a second set of transformed image pairs. In some of these variations, generating the set of stereo output images includes performing a video stabilization operation on the second set of transformed image pairs.

[0013] Generating the set of stereo output images may include generating metadata associated with the set of stereo output images. The metadata may include field of view information of at least one of the first camera or the second camera, pose information of the set of output images, and / or a set of default disparity values ​​for the set of stereo images in the stereo output images. In some cases, generating the metadata associated with the set of stereo output images includes selecting the set of default disparity values ​​based on a scene captured by the set of output images. In some variants, virtual content is added to the first set of transformed image pairs.

[0014] Some embodiments relate to a device comprising a first camera, a second camera, a set of displays, a memory, and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to perform any of the above methods. Similarly, other embodiments relate to a non-transitory computer-readable medium comprising instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any of the above methods.

[0015] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure will be readily understood through the following detailed description in conjunction with the accompanying drawings, in which like reference numerals represent like structural elements, and in which:

[0017] Figure 1 A block diagram of an exemplary electronic device that may be used in the extended reality systems described herein is shown.

[0018] Figure 2 An exemplary operating environment is shown that includes variations of electronic devices as described herein.

[0019] Figure 3 An example of an image stream is shown.

[0020] Figure 4 Depicted is a process of individually processing images in an image stream using a display image processing pipeline and an output image processing pipeline.

[0021] Figure 5 An example of a display image processing pipeline as described herein is depicted.

[0022] Figure 6 Depicted is an example of an output image processing pipeline as described herein.

[0023] Figure 7 Depicted are examples of image generation units that may be used as part of the output image processing pipeline described herein.

[0024] It should be understood that the proportions and sizes (relative or absolute) of the various features and elements (and collections and groupings thereof) and the boundaries, spacings, and positional relationships presented therebetween are provided in the accompanying drawings solely to facilitate understanding of the various embodiments described herein and therefore may not necessarily be presented or illustrated to measure and are not intended to indicate any preference or requirement for the embodiments illustrated to exclude the embodiments described in conjunction therewith. DETAILED DESCRIPTION

[0025] Reference will now be made in detail to the representative embodiments shown in the accompanying drawings. It should be understood that the following description is not intended to limit the embodiments to a preferred embodiment. On the contrary, it is intended to cover substitutions, modifications and equivalents that may be included in the essence and scope of the described embodiments defined by the appended claims.

[0026] Embodiments disclosed herein relate to devices, systems, and methods for processing an image stream separately to generate a first set of images for display by a set of displays and a second set of images to be stored or transmitted as part of a media capture event. Specifically, a first set of images from an image stream can be used to generate a first set of transformed images, and these transformed images can be displayed on a set of displays. A second set of images can be selected from the image stream in response to a capture request associated with the media capture event, and the second set of images can be used to generate a second set of transformed images. A set of output images can be generated from the second set of transformed images, and the set of output images can be stored or transmitted for later viewing.

[0027] Reference below Figures 1 to 7However, those skilled in the art will readily appreciate that the detailed description given herein with respect to these figures is for illustration purposes only and should not be construed as limiting.

[0028] The devices and methods described herein can be used as part of an extended reality system in which an extended reality environment is generated and displayed to a user. Various terms are used herein to describe the various extended reality systems and associated extended reality environments described herein. For example, as used herein, a "physical environment" is a portion of the physical / real world surrounding a user that the user can perceive and interact with without the assistance of an extended reality system described herein. For example, a physical environment can include a room or outdoor space in a building, as well as any person, animal, or object in that space (collectively referred to herein as a "real-world object"), such as plants, furniture, books, etc.

[0029] As used herein, an "extended reality environment" refers to a fully or partially simulated environment that a user can perceive and / or interact with using an extended reality system as described herein. In some cases, an extended reality environment may be a virtual reality environment, which refers to a fully simulated environment in which the user's physical environment is completely replaced by virtual content within the virtual reality environment. The virtual reality environment may not be dependent on the user's physical environment, and therefore may allow the user to perceive that they are in a different simulated location (e.g., standing at a beach when they are actually standing in a room of a building). The virtual reality environment may include virtual objects that the user can interact with (e.g., simulated objects that can be perceived by the user but do not actually exist in the physical environment).

[0030] In other cases, the extended reality environment may be a mixed reality environment, i.e., a fully or partially simulated environment in which virtual content may be presented together with a portion of the user's physical environment. Specifically, the mixed reality environment may include a reproduction and / or modified representation of one or more portions of the user's physical environment surrounding the extended reality system. In this way, the user is able to perceive their physical environment (directly or indirectly) through the mixed reality environment while still perceiving the virtual content.

[0031] As used herein, "reproduction" of a portion of a physical environment refers to a portion of an extended reality environment that recreates that portion of the physical environment within the extended reality environment. For example, an extended reality system may include one or more cameras capable of capturing images of the physical environment. The extended reality system may present a portion of these images to a user by displaying these images via an opaque display, so that the user indirectly views the physical environment via the displayed images. In addition, in some cases, an extended reality environment is displayed using foveated rendering, where different portions of the extended reality environment are rendered using different fidelity levels (e.g., image resolution) depending on the user's gaze direction. In these cases, for the purposes of this application, the reproduced portions rendered with lower fidelity using these foveated rendering techniques are still considered to be reproduced.

[0032] As used herein, a "modified representation" of a portion of a physical environment refers to a portion of an extended reality environment that is derived from the physical environment, but that intentionally obscures one or more aspects of the physical environment. While indirect reproduction attempts to replicate a portion of a user's physical environment within the extended reality environment, a modified representation intentionally changes one or more visual aspects of a portion of the user's physical environment (e.g., using one or more visual effects, such as artificial blur). In this way, a modified representation of a portion of a user's physical environment may allow the user to perceive certain aspects of that portion of the physical environment while obscuring other aspects. In an example of artificial blur, the user may still be able to perceive the general shape and placement of real-world objects within the modified representation, but may not be able to perceive visual details of those objects that are visible in the physical environment. Where foveated rendering is used to display the extended reality environment, portions of the modified representation that are in peripheral areas of the extended reality environment (relative to the user's gaze) may be rendered at a lower fidelity using foveated rendering techniques.

[0033] Reproductions and / or modified representations of a user's physical environment can be used in various extended reality environments. For example, an extended reality system may be configured to operate in a "pass-through" mode, during which the extended reality system generates and presents an extended reality environment that includes a reproduction of a portion of the user's physical environment. This allows the user to indirectly view a portion of their physical environment, even if one or more components of the extended reality system (e.g., a display) may interfere with the user's ability to directly view the same portion of their physical environment. When the extended reality system is operated in pass-through mode, some or all of the extended reality environment may include a reproduction of the user's physical environment. In some cases, in addition to the reproduction of the user's physical environment, one or more portions of the extended reality environment may include virtual content and / or a modified representation of the user's physical environment. Additionally or alternatively, virtual content (e.g., graphical elements of a graphical user interface, virtual objects) may be overlaid on the reproduced portion.

[0034] Generally, the extended reality systems described herein include an electronic device capable of capturing images and displaying the images to a user. Figure 1 A block diagram of an exemplary electronic device 100 is depicted, which can be part of an extended reality system as described herein. The electronic device 100 can be configured to capture, process, and display images as part of an extended reality environment. In some implementations, the electronic device 100 is a handheld electronic device (e.g., a smart phone or tablet) configured to present an extended reality environment to a user. In some of these implementations, the handheld electronic device can be temporarily attached to an accessory that allows the user to wear the handheld electronic device. In other implementations, the electronic device 100 is a head mounted device (HMD) worn by the user.

[0035] In some embodiments, the electronic device 100 has a bus 102 that operatively couples the I / O portion 104 to one or more computer processors 106 and memory 108, and includes circuits that interconnect components of the electronic device 100 and control communication between components. The I / O portion 104 includes various system components that can assist the operation of the electronic device 100. The electronic device 100 includes a set of displays 110 and a set of cameras 112. A set of cameras 112 can capture images of the user's physical environment (e.g., scene), and can use these images when generating an extended reality environment as described herein (e.g., as part of a pass-through mode). A set of displays 110 can display an extended reality environment so that the user can view the extended reality environment, thereby allowing the user to perceive its physical environment via the electronic device 100. A set of displays 110 may include a single display, or may include multiple displays (e.g., one display for each eye of the user). Each display in a set of displays may utilize any suitable display technology, and may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED) display, a quantum dot light emitting diode (QLED) display, etc.

[0036] The memory 108 of the electronic device 100 may include one or more non-transitory computer-readable storage devices. These non-transitory computer-readable storage devices may be used to store computer-executable instructions that, when executed by one or more computer processors 106, may cause the computer processors to perform the processes described herein (the various image capture and processing techniques described below). Additionally, the non-transitory computer-readable storage devices may be used to store images captured as part of a media capture event as described herein.

[0037] A computer-readable storage device may be any medium that may tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, apparatus, and device (e.g., one or more processors 106). In some examples, the storage device is a transient computer-readable storage medium. In some examples, the storage device is a non-transient computer-readable storage medium. Non-transient computer-readable storage devices may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like.

[0038] The one or more processors 106 may include, for example, a processor, a microprocessor, a programmable logic array (PLA), a programmable array logic (PAL), a general array logic (GAL), a complex programmable logic device (CPLD), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other programmable logic device (PLD) that can be configured to execute an operating system and applications for the electronic device 100 and facilitate the capture, processing, display, and storage of images, as described herein.

[0039] Thus, any of the processes described herein may be stored as instructions on a non-transitory computer-readable storage device so that a processor may utilize these instructions to perform various steps of the processes described herein. Similarly, the apparatus described herein includes a memory (e.g., memory 108) and one or more processors (e.g., processor 106) operatively coupled to the memory. The one or more processors may receive instructions from the memory and be configured to execute these instructions to perform various steps of the processes described herein. Any of the processes described herein may be performed using the apparatus described herein as a method of capturing and displaying an image.

[0040] In some implementations, the electronic device 100 includes a set of depth sensors 114 (which may include a single depth sensor or a plurality of depth sensors), each of which is configured to calculate depth information for a portion of the environment in front of the electronic device 100. Specifically, each of the depth sensors in the set of depth sensors 114 may calculate depth information within a coverage field (e.g., the widest lateral range at which the depth sensor can provide depth information). In some cases, the coverage field of a depth sensor in the set of depth sensors 114 may at least partially overlap with the field of view of at least one camera in the set of cameras 112 (e.g., the spatial range of a scene that a camera can capture using the image sensor of the camera), thereby allowing the depth sensor to calculate depth information associated with the field of view of the set of cameras 112.

[0041] Information from the depth sensor can be used to calculate the distance between the depth sensor and various points in the environment surrounding the electronic device 100. In some cases, the depth information from the set of depth sensors 114 can be used to generate a depth map, as described herein. The depth information can be calculated in any suitable manner. In one non-limiting example, the depth sensor can utilize stereo imaging, in which two images are taken from different positions, and the distance (difference) between corresponding pixels in the two images can be used to calculate depth information. In another example, the depth sensor can utilize structured light imaging, whereby the depth sensor can image the scene while projecting a known illumination pattern (typically using infrared illumination) toward the scene, and then can see how the pattern is distorted by the scene to calculate depth information. In yet another example, the depth sensor can utilize time-of-flight sensing, which calculates depth based on the amount of time it takes for light (typically infrared) emitted from the depth sensor to return from the scene. A time-of-flight depth sensor may utilize direct time-of-flight or indirect time-of-flight, and may illuminate the entire coverage field at once, or may illuminate only a subset of the coverage field at a given time (e.g., via one or more spots, strips, or other patterns that may be fixed or may scan across the coverage field). Where the depth sensor utilizes infrared illumination, the infrared illumination may be used for a range of environmental conditions without being perceived by the user.

[0042] In some implementations, the electronic device 100 includes an eye tracker 116. The eye tracker 116 may be configured to determine the position of the user's eyes relative to the electronic device 100 (or specific components thereof). The eye tracker 116 may include any suitable hardware for identifying and locating the user's eyes, such as one or more cameras, depth sensors, combinations thereof, etc. In some cases, the eye tracker 116 may also be configured to detect the position / direction of the user's gaze. It should be understood that the eye tracker 116 may include a single module configured to determine the position of the user's eyes, or may include multiple units, each of which is configured to determine the position of a corresponding eye of the user.

[0043] Additionally or alternatively, the electronic device 100 may include a set of sensors 118 capable of determining the motion and / or orientation of the electronic device 100 (or certain components thereof). Figure 1 The illustrated set of sensors 118 includes an accelerometer 120 and a gyroscope 122. Information from the set of sensors 118 may be used to compute pose information for the electronic device 100 (or specific components thereof). This may allow the electronic device 100 to track its position and orientation as the electronic device 100 moves. Additionally or alternatively, information captured by the set of cameras 112 and / or the set of depth sensors 114 may be used to compute pose information for the electronic device 100 (or specific components thereof), such as by using simultaneous localization and mapping (SLAM) techniques.

[0044] In addition, the electronic device 100 may include a set of input mechanisms 124 (e.g., a touch screen, soft keys, a keyboard, a virtual keyboard, buttons, knobs, joysticks, switches, dials, combinations thereof, etc.) that a user may manipulate to interact with and provide input to the augmented reality system. The electronic device 100 may include a communication unit 126 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. The electronic device 100 is not limited to Figure 1 Rather, other or additional components (e.g., GPS sensors, directional sensors such as a compass, physiological sensors, microphones, speakers, haptic engines, etc.) may be included in a variety of configurations as needed.

[0045] Figure 2 An operating environment 200 is shown that includes an electronic device 202, which can be part of an extended reality system to perform the processes described herein. The electronic device 202 can be described above with respect to Figure 1 The electronic device 202 may be configured in any manner as described for the electronic device 100. For example, the electronic device 202 may include: a set of cameras including a first camera 112a and a second camera 112b; and a set of displays including a first display 110a and a second display 110b. In addition, the electronic device 202 is shown to include a depth sensor 114 and an eye tracker 116, such as described herein.

[0046] The first camera 112a has a first field of view 230a and is positioned to capture an image of a first portion of the physical environment surrounding the electronic device 202. Similarly, the second camera 112b has a second field of view 230b and is positioned to capture an image of a second portion of the physical environment surrounding the electronic device 202. The first field of view 230a and the second field of view 230b may at least partially overlap, such that the first camera 112a and the second camera 112b may simultaneously capture a common portion of the physical environment surrounding the electronic device 202. For example, Figure 2 The operating environment 200 shown in FIG. 1 includes a real-world object 206 positioned in both a first field of view 230a and a second field of view 230b. Figure 2 When positioned as shown, images captured by both the first camera 112 a and the second camera 112 b will include the real-world object 206 .

[0047] As described in more detail herein, images captured by the first camera 112a and the second camera 112b may be displayed as part of an extended reality environment via a set of displays to provide the user with a representation of their physical environment (e.g., as part of a pass-through mode). For example, images captured by the first camera 112a may be displayed on the first display 110a (e.g., as part of an extended reality environment) and may be viewed by the user's first eye 204a, such as when the electronic device 202 is worn on the user's head or otherwise held near the user's head. Similarly, images captured by the second camera 112b may be displayed on the second display 110b (e.g., as part of an extended reality environment) and may be viewed by the user's second eye 204b.

[0048] Depending on the relative positioning between components of the electronic device 202 and the relative positioning between the electronic device 202 and the user's eyes 204a, 204b, the images captured by the first camera 112a and the second camera 112b may provide a viewpoint of the operating environment 200 that is different from the case where the user directly views their physical environment without the electronic device 202. Specifically, a set of cameras may be spatially offset from a set of displays in one or more directions. For example, the imaging plane of the first camera 112a may be separated from the first display 110a by a distance 220a along a first (e.g., horizontal) direction, and the imaging plane of the second camera 112b may be separated from the second display 110b by a distance 220b along the first direction. In some cases, the imaging planes of the first camera 112a and the second camera 112b may also be rotated relative to the first display 110a and the second display 110b, respectively.

[0049] In addition, the first eye 204a and the second eye 204b may be separated from the first display 110a and the second display 110b by corresponding distances (e.g., a first eye-to-display distance 222a between the first display 110a and the first eye 204a and a second eye-to-display distance 222b between the second display 110b and the second eye 204b). The first camera 112a is located by a first reference point (by Figure 2204a and 204b) are separated by a first scene distance 232a, and the second camera 112b is separated by a second reference point (which may be the same point as the first reference point or a different point). The offset (spatial and / or rotational) between the first camera 112a and the first display 110a, the first eye-to-display distance 222a, and the first scene distance 232a may collectively characterize the difference between the viewpoint of the first eye 204a and the viewpoint of the first camera 112a relative to the first reference point. Similarly, the offset between the second camera 112b and the second display 110b, the second eye-to-display distance 222b, and the second scene distance 232b may collectively characterize the difference between the viewpoint of the second eye 204b and the viewpoint of the second camera 112b relative to the second reference point.

[0050] Thus, it may be desirable to adjust images captured by a set of cameras to account for these viewpoint differences before the images are displayed (e.g., to generate perspective-corrected images), which may reduce discomfort to the user when viewing the images. However, these viewpoint adjustments may not be desirable in situations where a user wishes to capture images or videos using a set of cameras (e.g., to save for later viewing or sharing with others). These captured images may be viewed later by a different device (which may have a different configuration than device 202), viewed from a different viewpoint (e.g., viewed by a different user or when the user is at a different distance from the device), and / or may be viewed in a particular manner within an extended reality environment (e.g., viewed within a viewing window or range within the extended reality environment). Presenting perspective-corrected images in these situations may result in a poor user experience, and further adjusting these images to accommodate the new viewing environment may undesirably introduce or amplify artifacts.

[0051] The process described herein includes capturing data from one or more cameras (e.g., Figure 1 and Figure 2 The first image processing pipeline and the second image processing pipeline may receive the same image as input, but may utilize different image processing techniques customized for their respective outputs.

[0052] At least some of the images in the image collection are captured during a media capture event. Specifically, a set of cameras can be used to capture images during one or more photography modes (e.g., a photo mode that can capture still images, a video mode that can capture videos, a panoramic mode that can capture panoramic photos, a portrait mode that can capture still photos with artificial bokeh applied, etc.). Generally speaking, during these modes, the device can display (e.g., via Figure 1 and Figure 2 A camera user interface (a set of displays of the electronic device 100, 202) that displays a "live preview". The live preview is a stream of images captured by a set of cameras and presented in real time, and may represent the field of view that will be captured when the camera initiates a media capture event (which may be a subset of the camera's field of view). In other words, the live preview allows the user to see which part of the scene is currently being imaged and decide when to capture a photo or video.

[0053] The live preview may be presented to the user as part of an extended reality environment, where the extended reality environment is operating in a pass-through mode. In these cases, at least a portion of the extended reality environment includes a reproduction of the user's physical environment, thereby allowing the user to perceive their physical environment and understand which portion of the scene will be captured during the media capture event. The extended reality environment may include one or more graphical user elements or other features (e.g., transitions, such as transitions between a reproduction of the physical environment and a modified representation of virtual content or the physical environment that indicates the boundaries of the image to be captured by the device) to help guide the user in capturing the image.

[0054] When the extended reality system initiates a media capture event, the set of cameras will capture images and generate media depending on the current photography mode (e.g., capture and generate still photos when in photo mode, or capture and generate videos when in video mode), which media may then be stored locally on device 100 (e.g., stored in a non-transitory computer-readable storage device that is part of memory 108), transmitted to a remote server for storage, or transmitted to another system (e.g., as part of a messaging or streaming application). It should be understood that in some cases, there may be a frame buffer of images captured by the set of cameras, and some of the images collected before the media capture event is initiated may be used to generate the captured media that is stored or transmitted.

[0055] A media capture event may be initiated when an electronic device as described herein receives a capture request. A capture request may be received under some predetermined conditions (e.g., a software application running on a device may automatically request a set of cameras to initiate a media capture event when certain criteria are met, with appropriate user permission), or when a user gives a command to initiate a media capture event command by interacting with a shutter control on a user interface, pressing a designated button on an electronic device, giving a voice command, etc.

[0056] The processes described herein may be performed as part of a media capture event and may utilize a first image processing pipeline to generate a real-time preview image displayed to a user and a second image processing pipeline to generate captured media that is stored or transmitted. Figure 3 An exemplary image stream 300 is shown that includes a sequence of images of a scene captured by a set of cameras (e.g., Figure 1 and Figure 2 In an example where a set of cameras includes two cameras, the image stream 300 includes images captured by a first camera (e.g., Figure 2 The first camera image stream 300a of the image 302 captured by the first camera 112a of the electronic device 202 and the first camera image stream 300a of the image 302 captured by the second camera (eg, Figure 2 The second camera image stream 300b of the image 304 captured by the second camera 112b) of the electronic device 202.

[0057] A set of cameras may capture the image stream 300 continuously during operation of the electronic device, or may initiate the capture of the image stream 300 in response to certain conditions (e.g., the device enters a pass-through mode). When the image stream 300 includes multiple camera image streams (e.g., a first camera image stream 300a and a second camera image stream 300b), images from each of the camera streams (such as images captured simultaneously from different cameras) may be grouped and processed together. For example, the image stream 300 may include a series of image pairs, where each image pair includes a first image captured by a first camera (e.g., an image from the first camera image stream 300a) and a second image captured by a second camera (e.g., an image from the second camera image stream 300b).

[0058] For example, Figure 3The image stream 300 is shown to include a total of sixteen images (eight images 302 labeled L1-L8 in the first camera image stream 300a and eight images 304 labeled R1-R8 in the second camera image stream 300b), and will therefore have eight image pairs (the first pair of images includes images L1 and R1, the second pair of images includes images L2 and R2, and so on). A set of cameras may continue to capture the image stream 300 for as long as necessary to perform any media capture events that may be desired. A predetermined number of images (or image pairs) from the image stream 300 may be temporarily stored in a buffer at a given time (e.g., replacing the oldest image in the buffer with a new image), thereby allowing a certain number of recently captured images to be available to the device at any given time during the capture of the image stream 300.

[0059] The process described herein may select a first set 306 of images (or pairs of images) from the image stream 300, and will use the first set 306 of images to generate a set of real-time preview images, as described in more detail below. It should be understood that when the process as described herein is used to select a set of images for additional image processing (e.g., to generate images therefrom), the process need not wait for all images to be selected before initiating the image processing. Using the first set 306 of images as a non-limiting example, the device may begin generating real-time preview images from images L1 and R1 before other images from the first set 306 (such as images L7, L8, R7, and R8) have been captured. In other words, the first image processing pipeline may continuously generate real-time preview images.

[0060] like Figure 3As shown, a capture request 308 may be received during the capture of an image stream (e.g., shown here as being received during the capture of images L4 and R4). In response to receiving the capture request 308, a second set of images or image pairs 310 may be selected as part of a media capture event. The second set of images 310 may be used (e.g., by a second image processing pipeline) to generate a set of output images that form captured media that is stored or transmitted as discussed herein. In some cases, as previously described, the second set of images 310 may include one or more images or image pairs captured prior to the capture request 308 (e.g., images L3 and R3). The number of images 310 selected to be in the second set 310 may be predetermined (e.g., certain photography modes may select a specific number of images associated with each capture request 308) or dynamically determined (e.g., in a video mode, the number of images in the second set 310 may depend on when a termination request is received to stop the media capture event, thereby allowing a user to capture videos of different lengths). In some variations, receipt of capture request 308 may optionally prompt a group of cameras to temporarily change one or more operating parameters when capturing images. For example, in response to receiving capture request 308, a group of cameras may change the frame rate at which images are captured, or may intentionally change the exposure time of one or more images (e.g., the image pair including images L5 and R5 is shown as being captured with a longer exposure time than surrounding image pairs).

[0061] Some or all of the images used to generate output images as part of a media capture event (e.g., second set of images 310) may also be used to generate a real-time preview image for display to a user (e.g., via first set of images 306). Thus, different image processing pipelines may receive the same image as input, but will use different image processing techniques to generate multiple different images. For example, Figure 4 A variation of process 400 is depicted that uses multiple image processing pipelines to generate a first set of transformed images for display to a user (e.g., as part of a real-time preview in an extended reality environment) and a second set of transformed images for generating one or more output images as part of a media capture event. The blocks performed as part of process 400 may be performed as a method or may be stored as instructions on a non-transitory computer-readable storage device such that a processor may utilize the instructions to perform the various blocks of the processes described herein. Similarly, the devices described herein include a memory (e.g., memory 108) and one or more processors (e.g., processor 106) operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to perform the blocks of process 400.

[0062] Specifically, process 400 uses a set of cameras 404 to capture an image stream 402. In some cases, set of cameras 404 includes a first camera 404a and a second camera 404b, and image stream 402 includes image 402a captured by first camera 404a and image 402b captured by second camera 404b. In these cases, image stream 402 may form a series of image pairs, where each image pair includes a first image captured by first camera 404a and a second image captured by second camera 404b.

[0063] Process 400 includes a display image processing pipeline 406 configured to process images from an image stream 402 (such as, for example, from Figure 3 The display image processing pipeline 406 may generate a first set of transformed images 408 by using the first set 306 of images in the image stream 300. As part of generating the first set 408 of transformed images, the display image processing pipeline 406 may apply a first transformation operation to the images in the image stream 402. In some cases, the first transformation operation includes a perspective correction operation in which the images are transformed based on a difference between a viewpoint of one of the cameras in the set 404 and a viewpoint of a user. In these cases, the display image processing pipeline 406 may receive and utilize viewpoint data 410 that represents information about the viewpoint of the set 404 and the user. Figure 5 The perspective correction operation is described in more detail.

[0064] For each image in the image stream 402 input to the display image processing pipeline 406, the display image processing pipeline 406 may generate a corresponding transformed image in the first set of transformed images 408. For example, the image 402a captured by the first camera 404a may be used to generate a first subset 408a of these transformed images 408, and the image 402b captured by the second camera 404b may be used to generate a second subset 408a of the transformed images 408. Thus, the first set 408 of transformed images may include a first set of transformed image pairs, where each image pair in the image stream 402 is used to generate a corresponding transformed image pair in the first set of transformed image pairs.

[0065] At step 412, on a set of displays (e.g., Figure 1 The first set of transformed images can be displayed on a set of displays 110 of the electronic device 100. The first set of transformed images can be displayed as part of an extended reality environment (e.g., in a pass-through mode), which can provide a real-time preview such as discussed herein. Figure 2In the case of a first display 110a and a second display 110b of an electronic device 202 of the embodiment of the present invention, a first subset 408a of the transformed images 408 may be displayed on the first display, and a second subset 408b of the transformed images 408 may be displayed on the second display. In this way, the user's first eye may view the image captured by the first camera (via the first display), and the user's second eye may view the image captured by the second camera (via the second display). In the case where a set of displays includes a single display, each transformed image pair may be displayed simultaneously on the single display. For example, a first transformed image in a pair of images may be displayed on a first portion of the display (where the image is viewable by the user's first eye), while a second transformed image in the pair of images may be displayed on a second portion of the display (where the image is viewable by the user's second eye). In some cases, the display image processing pipeline 406 may add virtual content to the transformed images so that the user may view the virtual content when the transformed images are displayed to the user at block 412.

[0066] Additionally, a capture request may be received during the capture of the image stream 402. At block 414, a set of images 416 (e.g., from the image stream 300) is selected in response to receipt of the capture request. Figure 3 The set of selected images 416 includes one or more images captured after receiving the capture request, and in some cases may also include one or more images captured before receiving the capture request, such as described above with respect to Figure 3 The set 416 of selected images may include a first subset 416a of images captured by the first camera 404a (e.g., selected from the image 402a) and a second subset 416b of images captured by the second camera 404b (e.g., selected from the image 402b). Thus, the set 416 of selected images may include a selected set of image pairs, where each image pair includes a first image captured by the first camera 404a and a second image captured by the second camera 404b.

[0067] The process 400 may utilize an output image processing pipeline 418 configured to generate a set of output images 420 using the set of selected images 416. In some cases, the set of output images 420 includes a first subset of output images 420a generated from the first selected subset of images 416a and a second subset of output images 420b generated from the second subset of images 416b. Thus, the set of output images 420 may include a set of output image pairs. These output image pairs may include one or more stereo images and / or stereo videos. In stereo imaging, two images taken from different perspectives may allow the user to perceive depth when presented to different eyes of the user. Thus, a stereo image represents a single image pair that may be later presented to the user as a three-dimensional still image, while a stereo video represents a series of image pairs that may later be sequentially presented to the user as a three-dimensional video. Depending on the photography mode, the set of output images 420 may include a single stereo image, multiple stereo images (e.g., generated as part of a burst mode), only a stereo video, or a combination of a stereo image and a stereo video. Where the set of output images 420 includes both stereoscopic images and stereoscopic video, the stereoscopic images may be presented to the user as three-dimensional images, but the stereoscopic video clips associated with the stereoscopic images may also be played.

[0068] As part of generating the set of output images 420, the output image processing pipeline 418 may generate a second set of transformed images (not shown) from the set of selected images 416. The second set of transformed images may be used to generate the output images 420. In some cases, the output image processing pipeline 418 applies a second transform operation to the images in the image stream 402 that is different from the first transform operation applied by the display image processing pipeline 406. In this way, although the display image processing pipeline and the output image processing pipeline may sometimes receive the same images for processing, they will process these input images separately (e.g., in parallel) to provide different outputs. In some variations, the second transform operation includes an image correction operation, such as a first transform operation with respect to the image stream 402. Figure 6 Described in more detail.

[0069] In some cases, the output image processing pipeline 418 is also configured to generate metadata 422 associated with the set of output images 420. This metadata (an example of which will be described below with respect to Figure 7 4 (described in more detail) may be stored with the set of output images 420. In addition to the set of selected images 416, the output image processing pipeline 418 may also receive context information 424. The context information 424 may include information about the device (or its components, such as a set of cameras) and / or the scene when the image stream 404 was captured.

[0070] For example, in some variations, context information 424 includes a depth map 426. Depth map 426 includes a matrix of pixels, where each pixel corresponds to a corresponding location in a scene imaged by a set of cameras and has a corresponding depth value representing a distance between the device (or a portion thereof, such as a single camera in the set of cameras) and the corresponding location in the scene. In some cases, when a device includes a depth sensor (e.g., Figure 1 and Figure 2 In the case of a depth sensor 114 of an electronic device 100, 202, the depth map 426 may be derived from depth information generated by the depth sensor. Additionally or alternatively, the depth map 426 may be derived from image analysis of an image (or multiple images) captured by one or more cameras of the device (such as one or more images in the image stream 402). In these cases, monocular or stereo depth estimation techniques may be used to analyze the images to estimate depth information for different parts of the scene. It should also be understood that the output image processing pipeline 418 may receive multiple depth maps, such as a first depth map for a first camera 404a (which represents the scene distance relative to the first camera 404a) and a second depth map for a second camera 404b (which represents the scene distance relative to the second camera 404b).

[0071] Additionally or alternatively, context information 424 may include information about the content of the scene captured by set 416 of selected images. For example, context information 424 may include scene information map 428. Specifically, scene information map 428 includes a matrix of pixels, where each pixel corresponds to a corresponding position in the scene and has a corresponding identifier indicating the type of object present at the position (and optionally, a confidence value associated with the identifier). For example, scene information map 428 may be configured to identify the location of a scene including a face, such that each pixel of the scene information map indicates whether a face is present at the corresponding position in the scene, or includes a value (such as a normalized value between 0 and 1) indicating the confidence that a face is present at the corresponding position of the scene. In some variations, context information 424 may include multiple scene information maps (each scene information map is associated with a different type of object), or may include a single scene information map 428 that may indicate the presence of multiple types of objects (e.g., people, animals, trees, etc.) in the scene.

[0072] Specifically, the scene information map 428 (or multiple maps) can provide an understanding of the content of the scene, which can be used when generating the set of output images 420 and / or metadata 422. For example, the output image processing pipeline can be configured to apply different image processing techniques to people in the scene than to other objects present in the scene. Additionally or alternatively, the scene information map 428 (or multiple maps) can be used to select a default disparity value that is included in the metadata 422 and can be used later (e.g., when the output image is displayed) to control the disparity of the displayed output image (e.g., in a viewing area within an augmented reality environment).

[0073] The scene information map 428 may be generated in any suitable manner. For example, some or all of the set 416 of selected images and / or the depth map 426 may be analyzed, such as using image segmentation and / or image classification techniques, as will be readily understood by one of ordinary skill in the art. These techniques may be used to find a set of predetermined objects within a scene, the selection of which may be set based on user preferences, a photography mode that was active when the set 416 of selected images was captured, and / or a desired effect selected by the user to be applied to generate the set 420 of output images. In some variations, the scene information map 428 may be based at least in part on user input. For example, a user may identify an area of ​​interest of a scene (e.g., by determining a user's gaze as it relates to the scene, by determining a user's gesture as it relates to the scene, other user input, a combination thereof, etc.), and one or more objects may be identified in the area of ​​interest to generate the scene information map 428.

[0074] Additionally or alternatively, context information 424 may include gesture information 430 associated with the device (or one or more components thereof). Gesture information 430 may include the position and / or orientation of the device during the capture of the set of selected images 416. This may be used to generate similar gesture information associated with the set of output images 420. The gesture information associated with the set of output images 420 may be stored as metadata 422 and may be used later (e.g., when displaying the output images) to control the position and / or orientation at which the output images are displayed (e.g., in an extended reality environment).

[0075] At block 432, the set of output images 420 (including metadata 422 if the set of output images 420 is associated with such metadata 422) may be stored in a non-transitory computer-readable storage device (e.g., as a Figure 1As part of the memory 108 of the electronic device 100). Additionally or alternatively, the set of output images 420 may be transmitted to a different device (e.g., as part of an email, message, live broadcast, etc., or for remote storage). As part of storing and / or transmitting the set of output images 420, the set of output images 420 (including metadata 422 if the set of output images 420 is associated with such metadata 422) may be encoded or otherwise written in a specified format. As a non-limiting example, stereoscopic images may be written as a high-efficiency image file format, such as a high-efficiency image container (HEIC) file. As another non-limiting example, stereoscopic video may be written as a high-efficiency video coding (HEVC) file, such as a multi-view high-efficiency video coding (MV-HEVC) file.

[0076] By processing the image stream 402 separately between the display image processing pipeline 406 and the output image processing pipeline 418, these image processing pipelines can be customized to generate different types of images. Thus, the process can generate a first set of transformed images 408 at block 412 in a manner customized for real-time display of these images (e.g., as part of a real-time preview and / or pass-through mode in an augmented reality environment). These images can be discarded after they are presented to the user, or can optionally be stored in a non-transitory computer-readable storage device (e.g., as a Figure 1 4 (part of the memory 108 of the electronic device 100) as part of block 432. In addition, process 400 can generate a set 420 of output images that can be customized for different intended viewing experiences. For example, set 420 of output images can be intended to be viewed on a different device (having the same or a different configuration than the device used to perform process 400) and / or as part of a specific user experience having viewing requirements that are different from the viewing requirements of block 412.

[0077] It should be understood that the various blocks of process 400 may occur simultaneously and continuously. For example, certain images in first set 408 of transformed images may be displayed at block 412 while other images in first set 408 of transformed images are being generated by image processing pipeline 406. This may also occur while first camera 404a and second camera 404b are capturing images to be added to image stream 402 and while output image processing pipeline 418 is generating output image 420, as discussed herein.

[0078] Figure 5A variation of the display image processing pipeline operation 406 of process 400 is shown. As shown, the display image processing pipeline 406 includes a perspective correction operation 500 that generates a set of perspective-corrected images from a set of images received by the display image processing pipeline 406 (e.g., images in the image stream 402). In general, the perspective correction operation takes an image captured from one viewpoint (the "source viewpoint") and transforms the image so that it appears to be captured from a different viewpoint (the "target viewpoint"). Specifically, the perspective correction operation may utilize the user's viewpoint as the target viewpoint and transform the image accordingly so that it appears to be captured from the user's viewpoint. In this way, when the perspective-corrected images are displayed to the user at block 412 of process 400, objects presented in these images may appear at the same height and distance as real-world objects in the user's physical environment. This may improve the accuracy of reproducing the user's physical environment in an extended reality environment as part of a pass-through and / or live preview mode.

[0079] The perspective correction operation 500 may utilize the viewpoint data 410 to determine a source viewpoint (e.g., the viewpoint of a camera used to capture an image) and a target viewpoint (e.g., the viewpoint of a user or a particular eye of a user). For example, the viewpoint data 410 may include position information including one or more distances and / or angles representing spatial and / or rotational offsets between a set of cameras and a set of displays, respectively (such as, for example, with respect to Figure 2 In some cases, the offset between the set of cameras and the set of displays may be known a priori (e.g., when the set of cameras and the set of displays have a fixed spatial relationship) or may be determined dynamically by the device (e.g., when one or more of the set of cameras and / or the set of displays are movable within the device).

[0080] Viewpoint data 410 may include information about the spatial relationship between the user and the device (or its components). This may include a single set of values ​​representing the relative positioning of the user's eyes with respect to the device, or may include a different set of values ​​for each eye (such as, for example, a value for each eye). Figure 2 The relative positioning between the user and the device can be performed, for example, using an eye tracker such as Figure 1 and Figure 2 Similarly, viewpoint data 410 may include information about a scene imaged by a set of cameras (such as the distance from the device to a particular object in the scene). This may include information such as that obtained by a depth sensor (e.g., Figure 1 and Figure 2The distance information may be determined by a depth sensor 114 of the electronic device 100, 202, by one or more cameras in a group of cameras, by a combination thereof, etc.

[0081] Collectively, the viewpoint data 410 may be used to determine a difference between a source viewpoint and a target viewpoint for a particular image. The perspective correction operation 500 may use the difference between the source viewpoint and the target viewpoint to apply a transformation (e.g., a homography or a warping function) to an input image (e.g., an image in the image stream 402) to generate a perspective-corrected image. Specifically, for each pixel of the input image at a pixel position in an untransformed space, a new pixel position of the perspective-corrected image is determined in the transformed space of the transformed image. Examples of perspective correction operations are discussed in more detail in U.S. Patent No. 10,832,427 B1, entitled “Scene camera retargeting,” and U.S. Patent No. 11,373,271 B1, entitled “Adaptive image warping based on object and distance information,” the contents of which are hereby incorporated by reference in their entirety and are attached herein as appendices.

[0082] In the case where the first set of images includes a set of image pairs, the perspective correction operation 500 includes a first perspective correction operation 500a and a second perspective correction operation 500b. For each image pair, the first perspective correction operation 500a generates a first perspective-corrected image from the first image in the image pair. Similarly, the second perspective correction operation 500b generates a second perspective-corrected image from the second image in the image pair. Specifically, the first perspective correction operation 500a can be applied to the image 402a captured by the first camera 404a, and the perspective-corrected image can be generated based on the difference between the viewpoint of the first camera 404a and the viewpoint of the user (such as, for example, the viewpoint of the first eye of the user). Similarly, the second perspective correction operation 500b can be applied to the image 402b captured by the second camera 404b, and the perspective-corrected image can be generated based on the difference between the viewpoint of the second camera 404b and the viewpoint of the user (such as, for example, the viewpoint of the second eye of the user).

[0083] The perspective corrected images generated by the perspective correction operation 500 may form the first set of transformed images 408. In some cases, additional processing steps may be applied to these images. For example, at block 502, the display image processing pipeline 406 may add virtual content to the first set of transformed images 408, such as one or more graphical elements of a graphical user interface, a virtual object, etc. This may allow the process to generate an extended reality environment with virtual content, and may allow the user to experience the virtual content when the first set of transformed images 408 is displayed to the user at block 412 of the process 500. It should be understood that as part of generating the first set of transformed images 408, the display image processing pipeline 406 may perform additional processing steps, such as cropping a portion of the image.

[0084] Figure 6 A variation of the output image processing pipeline 418 of the process 400 is shown. As shown, the output image processing pipeline 418 includes a transformation unit 600 that performs a transformation operation on an image input into the output image processing pipeline 418 to generate a transformed image. Thus, the output image processing pipeline may be configured to generate a set of transformed images from a set 416 of selected images (e.g., a second set of transformed images that is different from the first set 408 of transformed images generated by the display image processing pipeline 406). The image generation unit 602 generates a set 420 of output images from the set of transformed images. The transformation operation performed by the transformation unit 600 may be different from the perspective correction operation 500 utilized in the display image processing pipeline operation 406. For example, in some variations, the transformation unit 600 applies an image correction operation that transforms the images to project them into a common target image plane. Figure 6 In the example of , the transformation unit 600 includes a dewarping operation 604 and an alignment operation 606, which together perform an image correction operation. In some cases, a set of cameras may have fisheye lenses or other similar optics designed to increase the field of view of these cameras. Although the images captured by these cameras can be useful for reproducing a three-dimensional environment (e.g., as part of an extended reality environment as described herein), these images often include distortions when viewed as two-dimensional images (e.g., object scales may change and straight lines in the physical environment may appear as curves in the captured images).

[0085] Thus, the dewarping operation 604 may generate a dewarped image to account for lens distortion, as will be readily understood by one of ordinary skill in the art. In other words, the image will be dewarped from its physical lens projection to a straight line projection, such that a straight line in a three-dimensional space is mapped to a straight line in a two-dimensional image. For each image in the set 416 of selected images received by the output image processing pipeline 418, a first dewarping operation 604a may generate a first dewarped image from a first image in an image pair (e.g., one of the first subset 416a of images captured by the first camera 404a), and a second dewarping operation 604a may generate a second dewarped image from a second image in the image pair (e.g., one of the second subset 416b of images captured by the second camera 404b). In the event that the first camera 404a and the second camera 404b have different configurations (e.g., different fields of view and / or different lens designs), the first dewarping operation 604a and the second dewarping operation 604b may be customized for the differences between these cameras.

[0086] For each image pair in the set of selected images 416, an alignment operation 606 may align the first dewarped image and the second dewarped image generated by the dewarping operation 604 so that the epipolar lines of these dewarped images are horizontally aligned. In this way, the transformation unit 600 may generate a transformed image pair for each image pair it receives that simulates having a common image plane (even though they may be captured by cameras with different image planes). These transformed image pairs may be used to generate the output image 420 via the image generation unit 602.

[0087] Figure 7 A variation of an image generation unit 602 is shown that is configured to generate a set 420 of output images as part of process 400. Optionally, image generation unit 602 may also be configured to generate metadata 422 associated with the set 420 of output images. Figure 7 As shown, the image generation unit 602 includes a still image processing unit 700 configured to generate a set of still images, a video processing unit 702 configured to generate a video, and a metadata processing unit 704 that generates metadata associated with the output images. For a given media capture event, the output image processing pipeline 418 will use some or all of these units when generating the set 420 of output images, depending on the photography mode.

[0088] The still image processing unit 700 is configured to receive a first set of transformed images 701 (which may be some or all of the transformed images generated by the transform unit 600), and may apply an image fusion operation 706 to generate one or more fused images 708 from two or more images in the first set of transformed images 701. In the case where the transformed images 701 include a set of transformed image pairs, the image fusion operation 706 generates one or more fused image pairs, which may form one or more fused stereo output images. In these cases, a first fused image 708a in the fused image pair may be generated by applying the first image fusion operation 706a to a first subset of the transformed images 701 (e.g., all of which are generated from images 416a captured by the first camera 404a). A second fused image 708b in the fused image pair may be generated by applying the second image fusion operation 706b to a second subset of the transformed images 701 (e.g., all of which are generated from images 416b captured by the second camera 404b). Typically, an image fusion operation includes selecting a reference image from a set of selected images and modifying some or all of its pixel values ​​using information from other selected images. The image fusion operation 706 may generate a fused output image with reduced noise and / or a higher dynamic range, and may utilize any suitable image fusion technique that will be readily understood by one of ordinary skill in the art.

[0089] The video processing unit 702 is configured to receive a second set of transformed images 703 (which may be some or all of the transformed images generated by the transform unit 600), and may perform a video stabilization operation 710 on the second set of transformed images 703 to generate a stabilized video 712. In the case where the second set of transformed images 703 includes a set of image pairs, the stabilized video 712 may be a stereoscopic video including a first set of images 712a (e.g., generated from images 416a captured by the first camera 404a) and a second set of images 712b (e.g., generated from images 416b captured by the second camera 404b). Thus, the video processing unit 702 may output a stereoscopic video. In the case where the set of output images 420 includes both fused output images and the stereoscopic video, the video processing unit 702 may optionally utilize one or more fused images 708 as part of the video stabilization operation 710 (e.g., frames of the stereoscopic video may be stabilized relative to the fused stereoscopic image). The video stabilization operation 710 may utilize any suitable electronic image stabilization technique that will be readily understood by one of ordinary skill in the art. In some cases, video stabilization operation 710 may receive contextual information 424 (eg, scene information, pose information, etc.), such as described above, which may assist in performing video stabilization.

[0090] In some variations, the image generation unit 602 may optionally include an additional content unit 730 configured to add additional content to the still images generated by the still image processing unit 700 and / or the videos generated by the video processing unit 702. For example, in some cases, the additional content unit 730 may apply virtual content 732 (e.g., one or more virtual objects) and / or other effects 734 (e.g., filtering, style transfer operations, etc.) to the output images generated by the image generation unit 602.

[0091] The metadata processing unit 704 is configured to generate metadata 422 associated with the set of output images 420. Depending on the type of metadata 422 being generated, the metadata processing unit 704 may receive a third set of images 705 (which may include some or all of the transformed images generated by the transform unit 600, the output images generated by the transform unit 600, and / or any of the images captured or generated as part of the process 400). Additionally or alternatively, the metadata processing unit 704 may receive context information 424, such as described herein, which may be used to generate the metadata 422.

[0092] In some variations, the metadata processing unit 704 is configured to generate field of view information 714 for use by a set of cameras to capture the set 416 of selected images for generating the set 420 of output images. The field of view information 714 may be fixed (e.g., where the field of view of the set of cameras is fixed), or may be dynamically determined (e.g., where one or more cameras in the set of cameras have optical zoom capabilities to provide a variable field of view). Additionally or alternatively, the metadata processing unit 704 is configured to generate baseline information 716 that represents one or more separation distances between cameras in the set of cameras (e.g., the separation distance between the first camera 404a and the second camera 404b). Again, this information may be fixed (e.g., where the set of cameras is in a fixed position relative to each other) or dynamically determined (e.g., where one or more cameras in the set of cameras may move with a device housing the set of cameras).

[0093] In some cases, metadata processing unit 718 is configured to generate pose information 718 associated with the set of output images. Specifically, pose information 718 may describe the pose of the output image, and thereby describe the relative position and / or orientation of the scene captured by set of output images 420. This information may include or otherwise be derived from pose information 430 associated with the underlying image (e.g., set of selected images 416) used to generate output image 420.

[0094] In some cases, the metadata processing unit 718 is configured to set a default disparity value 720 for the set of stereoscopic output images. The default disparity value for a given stereoscopic image represents a recommended disparity for rendering images in the stereoscopic image (e.g., the distance that the images are separated) when viewing the stereoscopic image. Changing the disparity at which the stereoscopic image is rendered will affect the way a user perceives the depth of the scene, and thus can be used to emphasize different parts of the scene captured by the set 420 of output images.

[0095] In some cases, it may be desirable to select a default set of disparity values ​​based on the scene captured by the set of output images 420. In these cases, context information 424, such as the scene information map 428, the depth map 426, and / or the pose information 430, may be used to select the default set of disparity values. The context information 424 may be associated with the set of output images 420 (e.g., generated from images or information captured simultaneously with the selected image 416 used to generate the set of output images 420). The context information 424 may allow the metadata processing unit 704 to determine one or more aspects of the scene captured by the set of output images 420 (e.g., the presence and / or location of one or more objects in the scene), and select a default disparity value based on these determinations.

[0096] The metadata 422 generated by the metadata processing unit 704 may include information applicable to all images in the set of output images and / or information specific to a particular image or pair of images. For example, the metadata 422 may include pose information 718 for an image or images in the output images 420, but may include a single set 716 of baseline information that applies to all output images 420. Similarly, the set of default disparity values ​​720 may include a single value that applies to each pair of images in the set 420 of output images, or may include different values ​​for different pairs of images (e.g., a first stereo image may have a first default disparity value and a second stereo image may have a second default disparity value).

[0097] For the purpose of illustration, the foregoing description uses specific nomenclature to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the embodiments. Therefore, for the purpose of illustration and description, the foregoing description of the specific embodiments described herein is presented. They are not intended to be exhaustive or to limit the embodiments to the disclosed precise forms. It will be apparent to those of ordinary skill in the art that, in view of the above teachings, many modifications and variations are feasible.

[0098] Implementation examples are described in the following numbered clauses:

[0099] 1. A device, comprising:

[0100] A set of cameras;

[0101] A set of displays;

[0102] Memory; and

[0103] One or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to:

[0104] capturing an image stream using the set of cameras;

[0105] generating a first set of transformed images from the image stream by applying a first transform operation to images in the image stream;

[0106] displaying the first set of transformed images using the set of displays;

[0107] receiving a capture request; and

[0108] In response to receiving the capture request, generating a set of stereoscopic output images, wherein generating the set of stereoscopic output images comprises:

[0109] selecting a set of images from the image stream;

[0110] generating a second set of transformed images from the set of images by applying a second transform operation different from the first transform operation to the set of images;

[0111] as well as

[0112] The set of stereo output images is generated using the second set of transformed images.

[0113] 2. The apparatus of clause 1, wherein generating the set of stereoscopic output images comprises:

[0114] A fused stereo output image is generated from two or more image pairs in the set of images.

[0115] 3. The apparatus of clause 1, wherein the set of stereoscopic output images comprises stereoscopic video.

[0116] 4. The apparatus of clause 3, wherein generating the set of stereoscopic output images comprises:

[0117] A video stabilization operation is performed on the second set of transformed images.

[0118] 5. Apparatus according to any of clauses 1-4, wherein the first transformation operation is a perspective correction operation.

[0119] 6. Apparatus according to any of clauses 1-5, wherein the second transformation operation is an image correction operation.

[0120] 7. Apparatus according to any of clauses 1-6, wherein generating the set of stereoscopic output images comprises generating metadata associated with the set of stereoscopic output images.

[0121] 8. The apparatus of clause 7, wherein the metadata comprises field of view information of the set of cameras.

[0122] 9. An apparatus according to clause 7 or clause 8, wherein the metadata comprises pose information for the set of stereoscopic output images.

[0123] 10. Apparatus according to any of clauses 7-9, wherein the metadata comprises a default set of disparity values ​​for the set of stereoscopic output images.

[0124] 11. The apparatus of clause 10, wherein generating metadata associated with the set of stereoscopic output images comprises selecting the set of default disparity values ​​based on a scene captured by the set of stereoscopic output images.

[0125] 12. A method comprising:

[0126] Use a device's set of cameras to capture image streams;

[0127] generating a first set of transformed images from the image stream by applying a first transform operation to images in the image stream;

[0128] displaying the first set of transformed images at a set of displays of a device;

[0129] receiving a capture request; and

[0130] In response to receiving the capture request, a set of stereoscopic output images is generated, wherein

[0131] Generating the set of stereo output images comprises:

[0132] selecting a set of images from the image stream;

[0133] generating a second set of transformed images from the set of images by applying a second transform operation different from the first transform operation to the set of images; and

[0134] The set of stereo output images is generated using the second set of transformed images.

[0135] 13. The method of clause 12, wherein generating the set of stereoscopic output images comprises:

[0136] A fused stereo output image is generated from two or more image pairs in the set of images.

[0137] 14. The method of clause 12, wherein the set of stereoscopic output images comprises stereoscopic video.

[0138] 15. The method of clause 14, wherein generating the set of stereoscopic output images comprises:

[0139] A video stabilization operation is performed on the second set of transformed images.

[0140] 16. A method according to any of clauses 12-15, wherein the first transformation operation is a perspective correction operation.

[0141] 17. A method according to any of clauses 12-16, wherein the second transformation operation is an image correction operation.

[0142] 18. A method according to any of clauses 12-17, wherein generating the set of stereoscopic output images comprises generating metadata associated with the set of stereoscopic output images.

[0143] 19. The method of clause 18, wherein the metadata comprises field of view information of the set of cameras.

[0144] 20. A method according to clause 18 or clause 19, wherein the metadata comprises pose information for the set of stereoscopic output images.

[0145] 21. A method according to any of clauses 18-20, wherein the metadata comprises a set of default disparity values ​​for the set of stereoscopic output images.

[0146] 22. The method of clause 21, wherein generating metadata associated with the set of stereoscopic output images comprises selecting the set of default disparity values ​​based on a scene captured by the set of stereoscopic output images.

[0147] 23. A method according to any of clauses 12-22, comprising adding virtual content to the first set of transformed images.

[0148] 24. A non-transitory computer-readable medium comprising instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any of clauses 12-23.

[0149] 25. A device, comprising:

[0150] First Camera;

[0151] Second camera;

[0152] A set of displays;

[0153] Memory; and

[0154] one or more processors, the one or more processors being operatively coupled to the memory, wherein the one or more processors are configured to execute so that the one or

[0155] Instructions for multiple processors to:

[0156] capturing a first set of image pairs of a scene using the first camera and the second camera, wherein each image pair in the first set of image pairs comprises:

[0157] a first image captured by the first camera; and a second image captured by the second camera;

[0158] Generating a first set of transformed image pairs from the first set of image pairs, wherein generating the first set of transformed image pairs comprises, for each image pair in the first set of image pairs:

[0159] generating a first perspective-corrected image from the first image based on a difference between a viewpoint of the first camera and a viewpoint of a user; and generating a second perspective-corrected image from the second image based on a difference between a viewpoint of the second camera and the viewpoint of the user;

[0160] displaying the first set of transformed image pairs on the set of displays;

[0161] receiving a capture request;

[0162] In response to receiving the capture request, selecting a second set of image pairs from the first set of image pairs;

[0163] Generating a second set of transformed image pairs from the second set of image pairs, wherein generating the second set of transformed image pairs comprises, for each image pair in the first set of image pairs:

[0164] generating a first dewarped image from the first image;

[0165] generating a second dewarped image from the second image; and aligning the first dewarped image with the second dewarped image;

[0166] as well as

[0167] A set of output images is generated using the second set of transformed image pairs.

[0168] 26. The apparatus of clause 25, wherein generating the set of output images comprises:

[0169] A fused stereo output image is generated from two or more image pairs in the second set of transformed image pairs.

[0170] 27. The apparatus of clause 25, wherein the set of output images comprises a stereoscopic video formed from the second set of transformed image pairs.

[0171] 28. The apparatus of clause 27, wherein generating the set of output images comprises:

[0172] A video stabilization operation is performed on the second set of transformed image pairs.

[0173] 29. Apparatus according to any of clauses 25-28, wherein generating the set of output images comprises generating metadata associated with the set of output images.

[0174] 30. The apparatus of clause 29, wherein the metadata comprises field of view information of at least one of the first camera or the second camera.

[0175] 31. An apparatus according to clause 29 or clause 30, wherein the metadata includes pose information for the set of output images.

[0176] 32. Apparatus according to any of clauses 29 to 31, wherein:

[0177] The set of output images comprises a set of stereoscopic images; and

[0178] The metadata comprises a set of default disparity values ​​for the set of stereoscopic output images.

[0179] 33. The apparatus of clause 32, wherein generating metadata associated with the set of output images comprises selecting the set of default disparity values ​​based on the scene captured by the set of output images.

[0180] 34. Apparatus according to any of clauses 25-33, wherein the processor is configured to add virtual content to the first set of transformed image pairs.

[0181] 35. A method comprising:

[0182] A first set of image pairs of a scene are captured using a first camera and a second camera of the device.

[0183] A method for preparing an image pair comprising:

[0184] a first image captured by the first camera; and

[0185] a second image captured by the second camera;

[0186] Generating a first set of transformed image pairs from the first set of image pairs, wherein generating the first set of transformed image pairs comprises:

[0187] For each image pair:

[0188] generating a first perspective-corrected image from the first image based on a difference between a viewpoint of the first camera and a viewpoint of a user; and

[0189] generating a second perspective-corrected image from the first image based on a difference between a viewpoint of the second camera and the viewpoint of the user;

[0190] displaying a first set of transformed image pairs on a set of displays of the device;

[0191] receiving a capture request;

[0192] In response to receiving the capture request, selecting a second set of image pairs from the first set of image pairs;

[0193] Generating a second set of transformed image pairs from the second set of image pairs comprises

[0194] For each image pair in the first set of image pairs:

[0195] generating a first dewarped image from the first image;

[0196] generating a second dewarped image from the second image; and

[0197] aligning the first dewarped image and the second dewarped image; and generating a set of output images using the second set of transformed image pairs.

[0198] 36. The method of clause 35, wherein generating the set of output images comprises:

[0199] A fused stereo output image is generated from two or more image pairs in the second set of transformed image pairs.

[0200] 37. The method of clause 35, wherein the set of output images comprises a stereoscopic video formed by the second set of transformed image pairs.

[0201] 38. The method of clause 37, wherein generating the set of output images comprises:

[0202] A video stabilization operation is performed on the second set of transformed image pairs.

[0203] 39. A method according to any of clauses 35-38, wherein generating the set of output images comprises: generating metadata associated with the set of output images.

[0204] 40. The method of clause 39, wherein the metadata comprises field of view information of at least one of the first camera or the second camera.

[0205] 41. A method according to clause 39 or clause 40, wherein the metadata includes pose information for the set of output images.

[0206] 42. A method according to any of clauses 39 to 41, wherein:

[0207] The set of output images comprises a set of stereoscopic images; and

[0208] The metadata comprises a set of default disparity values ​​for the set of stereoscopic output images.

[0209] 43. The method of clause 42, wherein generating metadata associated with the set of output images comprises selecting the set of default disparity values ​​based on the scene captured by the set of output images.

[0210] 44. A method according to any of clauses 35-43, comprising adding virtual content to the first set of transformed image pairs.

[0211] 45. A non-transitory computer-readable medium comprising instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any of clauses 35-44.

Claims

1. A device, comprising: A set of cameras; A set of displays; Memory; and One or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to: capturing an image stream using the set of cameras; generating a first set of transformed images from the image stream by applying a first transform operation to images in the image stream; displaying the first set of transformed images using the set of displays; receiving a capture request; as well as In response to receiving the capture request, generating a set of stereoscopic output images, The step of generating the set of stereo output images comprises: selecting a set of images from the image stream; generating a second set of transformed images from the set of images by applying a second transform operation different from the first transform operation to the set of images; and The set of stereo output images is generated using the second set of transformed images.

2. The apparatus of claim 1 , wherein generating the set of stereo output images comprises: A fused stereo output image is generated from two or more image pairs in the image set.

3. The apparatus of claim 1, wherein the set of stereoscopic output images comprises stereoscopic video.

4. The apparatus of claim 3, wherein generating the set of stereoscopic output images comprises: A video stabilization operation is performed on the second set of transformed images. The apparatus of claim 1 , wherein the first transformation operation is a perspective correction operation. The apparatus of claim 1 , wherein the second transformation operation is an image correction operation.

7. The apparatus of claim 1 , wherein generating the set of stereoscopic output images comprises: Metadata associated with the set of stereoscopic output images is generated. The apparatus of claim 7 , wherein the metadata comprises field of view information of the set of cameras.

9. The apparatus of claim 7, wherein the metadata comprises pose information for the set of stereoscopic output images.

10. The apparatus of claim 7, wherein the metadata comprises a set of default disparity values ​​for the set of stereoscopic output images.

Citation Information

Patent Citations

  • Scene camera retargeting

    US10832427B1

  • Adaptive image warping based on object and distance information

    US11373271B1

  • Camera rig and stereoscopic image capture

    US10244226B2

  • System configuration for peripheral vision with reduced size, weight, and cost

    US11206364B1

  • Systems and methods for generating stabilized images of a real environment in artificial reality

    US11481960B2