Image processing with adjustable background blurring effects

Abstract

This disclosure provides systems, methods, and devices for image signal processing that support adjusting background blurring effects of image data. In a first aspect, a method of image processing includes receiving a first user input specifying a first depth of focus. The camera is configured with a first aperture size of a plurality of aperture sizes of the camera that is closest to a second aperture size corresponding to the first focal depth. The first aperture size is smaller than the second aperture size. Image data representing a scene is received, the image data being captured with a first aperture size of a camera. First modified image data is determined by applying a first blur kernel to the image data. The first modified image data represents a scene at a depth of focus corresponding to a second aperture size. Other aspects and features are also claimed and described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. patent application Ser. No. 18 / 180,593, filed on Mar. 8, 2023, entitled “IMAGE PROCESSING WITH ADJUSTABLE BOKEH EFFECT,” which is incorporated herein by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure generally relate to image processing, and more particularly, to adjusting the background blur (Bokeh) effect of an image. Some features may achieve and provide improved image processing, including generating a background blur effect for an image corresponding to an aperture size of a camera that is different from the aperture size of the camera's captured image. Background Art

[0004] An image capture device is a device that can capture one or more digital images, whether still images for photographs or image sequences for videos. Capture devices can be incorporated into a variety of devices. By way of example, image capture devices may include standalone digital cameras or digital video cameras, camera-equipped wireless communication devices (such as mobile phones, cellular or satellite radios), personal digital assistants (PDAs), tablets or tablets, gaming devices, computing devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0005] In some scenes, the photographer may wish to direct the viewer's focus to a specific part of the scene. For example, in a portrait, the photographer may want the viewer to focus on the person rather than the surrounding scenery. For such photos, the photographer may choose a low-aperture lens because the low aperture causes objects at a different depth than the person to be noticeably blurred. Compared to high-aperture lenses, low-aperture lenses produce greater blur. However, low-aperture lenses are typically larger and made of more expensive materials. Summary of the Invention

[0006] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all anticipated features of the present disclosure and is not intended to identify key or important elements of all aspects of the present disclosure, nor is it intended to delineate the scope of any or all aspects of the present disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to the more detailed description that will be given later.

[0007] An image representing a scene and displaying a background blur effect includes portions of the scene in the image that are in focus and portions of the scene in the image that are blurred and out of focus. The aperture size of the lens capturing the image may determine which portions of the image are in focus and which portions are out of focus, and / or may determine the level of blurriness of the out-of-focus portions. An image processing method is provided for adjusting a background blur effect displayed in an image representing a scene by adjusting which portions of the scene in the image are blurred versus which portions are in focus and / or adjusting the level of blurriness of certain portions of the scene in the image. In some embodiments, an image processing method is provided for adjusting the background blur effect when the image is captured using a variable aperture camera. In other embodiments, an image processing method is provided for adjusting the background blur effect when the image is captured using an immutable (e.g., fixed) aperture camera.

[0008] In one aspect of the present disclosure, an image processing method includes: receiving first user input specifying a first focal depth; configuring a camera to have a first aperture size of a plurality of aperture sizes of the camera, the first aperture size being closest to a second aperture size corresponding to the first focal depth, wherein the first aperture size is smaller than the second aperture size; receiving image data, the image data representing a scene captured using the first aperture size of the camera; and determining first modified image data by applying a first blur kernel to the image data, wherein the first modified image data represents the scene at the focal depth corresponding to the second aperture size.

[0009] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to: perform operations including receiving a first user input specifying a first depth of focus; configuring a camera to have a first aperture size of a plurality of aperture sizes of the camera that is closest to a second aperture size corresponding to the first depth of focus, wherein the first aperture size is smaller than the second aperture size; receiving image data representing a scene captured using the first aperture size of the camera; and determining first modified image data by applying a first blur kernel to the image data, wherein the first modified image data represents the scene at the depth of focus corresponding to the second aperture size.

[0010] In an additional aspect of the present disclosure, an image processing method includes receiving image data, the image data including a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus, the image data being captured by a camera at a first aperture size; receiving a first user input specifying a first depth of focus corresponding to a second aperture size different from the first aperture size; and when the second aperture size is less than the first aperture size: determining a combined image frame including at least a first portion of the first image frame and a second portion of the second image frame, wherein the first portion corresponds to the first focus and the second portion corresponds to the second focus; and determining first modified image data from the combined image frame by applying a first blur kernel to the combined image frame. The first modified image data represents the scene at the depth of focus corresponding to the second aperture size.

[0011] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to: perform operations including receiving image data, the image data including a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus, the image data being captured by a camera at a first aperture size; receiving first user input specifying a first depth of focus corresponding to a second aperture size different from the first aperture size; and when the second aperture size is smaller than the first aperture size: determining a combined image frame including at least a first portion of the first image frame and a second portion of the second image frame, wherein the first portion corresponds to the first focus and the second portion corresponds to the second focus; and determining first modified image data from the combined image frame by applying a first blur kernel to the combined image frame. The first modified image data represents the scene at the depth of focus corresponding to the second aperture size.

[0012] The image processing methods described herein can be performed by an image capture device and / or on image data captured by one or more image capture devices. Image capture devices (devices that can capture one or more digital images, whether still images, photographs, or video sequences) can be incorporated into a variety of devices. By way of example, image capture devices can include standalone digital cameras or digital video cameras, camera-equipped wireless communication devices (such as mobile phones, cellular or satellite radios), personal digital assistants (PDAs), tablets or tablets, gaming devices, computing devices (such as web cameras, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0013] The image processing techniques described herein may involve a digital camera having an image sensor and processing circuitry (e.g., an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), or a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and be configured to perform operations to obtain image data for processing according to the image processing techniques described herein and / or referred to in the image processing techniques described herein. The ISP may be configured to control the capture of image frames from one or more image sensors and determine one or more image frames from the one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames forming a video sequence. The video sequence may include other image frames received from the image sensor or other image sensors.

[0014] In an example application, an image signal processor (ISP) may receive instructions to capture a sequence of image frames in response to the loading of software (such as a camera application) to generate a preview display from an image capture device. The ISP may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from the image sensors, merged image data from the image sensors, or corrected image data processed by one or more algorithms within the ISP. For example, the image frames may be processed by an image post-processing engine (IPE) and / or other image processing circuitry to process the image frames obtained from the image sensors (which may have undergone some processing prior to output to the ISP) in the ISP to perform one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, and the like. The output image frames from the ISP may be stored in memory and retrieved by an application processor executing the camera application. The application processor may perform further processing on the output image frames to adjust their appearance and reproduce them on a display for viewing by a user.

[0015] After the output image frame representing the scene is determined by the image signal processor and / or application processor (such as by the image processing techniques described in various embodiments herein), the output image frame can be displayed on the device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, an image signal processor (ISP) can be configured to obtain input frames of image data (e.g., pixel values) from one or more image sensors and then generate corresponding output image frames (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor can output the image frames to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., autofocus (AF), auto white balance (AWB), and automatic exposure control (AEC)), generating a video file via the output frames, configuring the frames for display, configuring the frames for storage, sending the frames over a network connection, etc. Generally speaking, an image signal processor (ISP) may obtain incoming frames from one or more image sensors and generate and output a stream of output frames to various output destinations.

[0016] In some aspects, the output image frame can be generated by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, the first image frame and the second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that can result in improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method can be performed for MFNR photography, where the first image frame and the second image frame are captured using the same or different exposure times and are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0017] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a binning module or otherwise controlling aspects of image correction. The methods and techniques described herein can be performed entirely by the image signal processor or processor, or various operations can be split between the image signal processor and the processor, and in some aspects, across additional processors.

[0018] The device may include one, two, or more image sensors, such as a first image sensor. When multiple image sensors are present, the configurations of these image sensors may vary. For example, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire images through a first lens having a first optical axis, and the second sensor is configured to acquire images through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. Any of these or other configurations may be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back of the mobile device. Additional image sensors may be included with larger, smaller, or the same field of view. The image processing techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0019] In additional aspects of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes components for capturing image frames. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge-coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), and a time-of-flight detector. The device may also include one or more components for collecting and / or focusing light onto one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame that are input to the image processing techniques described herein.

[0020] For those of ordinary skill in the art, when reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings, other aspects, features and specific implementations will become apparent. Although each feature may be discussed below with respect to certain aspects and drawings, various aspects may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to various aspects. In a similar manner, although exemplary aspects may be discussed below as devices, systems or methods, exemplary aspects may be implemented in various devices, systems and methods.

[0021] The method may be embodied as computer program code on a computer-readable medium, the computer program code comprising instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device comprising: a first network adapter configured to transmit data, such as recorded images or video, or streaming data, via a first network connection among a plurality of network connections; and a processor coupled to the first network adapter and a memory. The processor may cause the output image frames described herein to be transmitted via a wireless communication network, such as a 5G NR communication network.

[0022] The features and technical advantages of examples according to the present disclosure have been outlined rather broadly above so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The concepts and specific examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and method of operation) and the associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of the figures in the drawings is provided for the purpose of illustration and description and not as a definition of limitations of the claims.

[0023] While various aspects and implementations are described herein through the lens of a few examples, those skilled in the art will appreciate that additional implementations and use cases may arise in many different arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, various aspects and / or uses may arise via integrated chip implementations and other non-module component-based devices (e.g., end-user devices, vehicles, communications equipment, computing devices, industrial equipment, retail / purchase equipment, medical devices, artificial intelligence (AI)-enabled devices, etc.). While some examples may or may not be specifically directed to use cases or applications, a wide range of applicability of the described innovations may be apparent. Implementations may range from chip-level or module components to non-module, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more aspects of the described innovations. In some practical environments, devices incorporating the described aspects and features may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals necessarily include multiple components for both analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A further understanding of the nature and advantages of the present disclosure may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a dash and a second reference number to distinguish between similar components. If only the first reference number is used in the specification, the description applies to any of the similar components having the same first reference number, regardless of the second reference number.

[0025] Figure 1 A block diagram of an example device for performing image capture from one or more image sensors is shown.

[0026] Figure 2 is a block diagram illustrating example data flow paths for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0027] Figure 3 A flow chart illustrating an example method for processing image data to adjust a background blur effect for image data captured using a variable aperture lens, according to some embodiments of the present disclosure.

[0028] Figure 4 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0029] Figure 5 A flow chart illustrating an example method for processing image data to adjust a background blur effect for image data captured with a non-variable aperture lens, according to some embodiments of the present disclosure.

[0030] Figure 6 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0031] Figure 7 It is a depiction of the scene represented by image data.

[0032] Figure 8 is a block diagram illustrating an image processing technique for determining an image frame with an adjustable background blur effect according to some embodiments of the present disclosure.

[0033] The same reference numbers and names in different drawings represent the same elements. DETAILED DESCRIPTION

[0034] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Instead, the detailed description includes specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0035] The present disclosure provides systems, devices, methods, and computer-readable media that support image processing, including techniques for adjusting a background blur effect displayed in an image representing a scene by adjusting which portions of the scene in the image are blurred versus which portions are in focus and / or adjusting the blurriness level of certain portions of the scene in the image. In some embodiments, an image processing method for adjusting the background blur effect when capturing an image using a variable aperture (VA) camera is provided. In an example of such an embodiment, a user-selected depth of focus for image capture using the VA camera is received via an input function on a device including the VA camera. The aperture size of the VA camera that is closest to and smaller than the simulated aperture size corresponding to the selected depth of focus is selected. An image can then be captured using the VA camera set to the selected aperture size. The captured image is modified by applying a blur kernel to the captured image so that, although the VA camera does not actually include an aperture size equal to the simulated aperture size, the resulting modified image still represents an image captured using the simulated aperture size.

[0036] In other embodiments, an image processing method for adjusting a background blur effect when capturing an image using an immutable (e.g., fixed) aperture camera is provided. In such other embodiments, a user-selected depth of focus for image capture using a fixed aperture (FA) camera is received via an input function on a device including the FA camera. The selected depth of focus corresponds to a simulated aperture size that is smaller than the aperture size of the FA camera. An image can then be captured using the FA camera. The captured image is modified so that every region of a scene represented by the captured image is in focus, such that a fully focused image is determined by the modification. The fully focused image is modified by applying a blur kernel to the fully focused image so that, although the FA camera does not actually include an aperture size equal to the simulated aperture size, the resulting modified image still represents an image captured using the simulated aperture size.

[0037] Certain implementations of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for generating an improved bokeh effect in an image for different capture conditions for either VA or non-VA cameras. In some aspects, techniques are provided for enabling a user to select a focus target for a bokeh effect in an image after the image is captured. In some aspects, techniques are provided for enabling a user to adjust the blurriness level of one or more objects in an image (e.g., the background of the image) after the image is captured.

[0038] Example devices for capturing image frames using one or more image sensors, such as smartphones, may include a configuration of one, two, three, four, or more cameras on the back side (e.g., the side opposite the primary user display) and / or the front side (e.g., the same side as the primary user display) of the device. These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frames in a memory and / or provide the output image frames to processing circuitry in other ways (e.g., via a bus). The processing circuitry may perform further processing, such as encoding, storing, transmitting, or other manipulation of the output image frames.

[0039] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor that generate image frames for processing by an image signal processor or other logic circuitry or for storage in memory (whether a short-term buffer or long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing the individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to digital circuitry coupled to the image sensor.

[0040] In the description of the embodiments herein, a large number of specific details (such as examples of specific components, circuits and processes) are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that these specific details may not be needed to implement the teachings disclosed herein. In other examples, in order to avoid blurring the teachings of the present disclosure, well-known circuits and devices are shown in block diagram form.

[0041] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc. is conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulation of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0042] In the accompanying drawings, a single block may be described as performing one or more functions. The one or more functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Those skilled in the art may implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of this disclosure. Moreover, the example device may include components other than those shown, including well-known components such as processors, memories, etc.

[0043] Aspects of the present disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or "frames"). The terms "output image frame" and "corrected image frame" may refer to image frames that have been processed by any of the techniques discussed. Furthermore, aspects of the present disclosure may be implemented in image sensors or devices coupled to such image sensors having the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). Furthermore, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including processing devices residing in a cloud computing system.

[0044] Unless otherwise specifically noted, as will be apparent from the following discussion, it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0045] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some portions of the present disclosure. Although the description and examples herein use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus can include a device or a portion of a device for performing the described operations.

[0046] Certain components of a device or apparatus described as “means for accessing,” “means for receiving,” “means for transmitting,” “means for using,” “means for selecting,” “means for determining,” “means for normalizing,” “means for multiplying,” or other similarly named terms referring to one or more operations on data (such as image data) may refer to processing circuitry (e.g., application specific integrated circuit (ASIC), digital signal processor (DSP), graphics processing unit (GPU), central processing unit (CPU)) configured to perform the described functions through hardware, software, or a combination of hardware configured by software.

[0047] Figure 1 A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. Device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. Device 100 may also include or be coupled to a display 114 and input / output (I / O) components 116. I / O components 116 may be used for user interaction, such as a touchscreen interface and / or physical buttons. Processor 104 and / or image signal processor 112 may perform operations to control cameras 103 and 105, such as adjusting the size of a variable aperture and / or adjusting a background blur effect in image data captured by cameras 103 and 105.

[0048] I / O components 116 may also include network interfaces for communicating with other devices, including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and / or 154 may be coupled to an antenna, including multiple antennas configured for primary and diversity reception and / or configured to receive specific frequency bands.

[0049] The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to Figure 1 Additional features or components not shown in the figure. In one example, a wireless interface, which may include multiple transceivers and a baseband processor, may be coupled to or included in WAN adapter 152 for wireless communication. In another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between image sensors 101 and 102 and image signal processor 112.

[0050] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data regarding the movement of the device 100, data regarding the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which can also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, with one or more of the acceleration, velocity, and / or distance being included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0051] Image signal processor 112 can receive image data, such as for use in forming image frames. In one embodiment, a local bus connection couples image signal processor 112 to image sensors 101 and 102 of first camera 103 and second camera 105, respectively. In another embodiment, a wired interface couples image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples image signal processor 112 to image sensors 101 and 102.

[0052] The first camera 103 may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated autofocus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0053] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. Lenses 131 and 132, respectively, focus light onto image sensors 101 and 102 using one or more apertures for receiving light, one or more shutters for blocking light outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 can have different fields of view to capture different representations of a scene. For example, the first lens 131 can be an ultra-wide (UW) lens, and the second lens 132 can be a wide (W) lens. The multiple image sensors can include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors.

[0054] That is, each image sensor can be configured through hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensors are configured with different lenses with different magnifications, which result in different fields of view. The sensors can be configured so that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV can capture a field of view in the range of 64 degrees to 84 degrees, a sensor configured for an ultra-lateral FOV can capture a field of view in the range of 100 degrees to 140 degrees, a sensor configured for a long-range FOV can capture a field of view in the range of 10 degrees to 30 degrees, and a sensor configured for an ultra-long-range FOV can capture a field of view in the range of 1 degree to 8 degrees.

[0055] Camera 103 may have a variable aperture (VA) camera, where the aperture can be controlled to a specific size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. Camera 103 may have different characteristics based on the current aperture size, such as different depths of focus (DOF) at different aperture sizes.

[0056] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Figure 1 Device 100 is illustrated as including two image sensors 101 and 102 coupled to image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor, such as depth sensor 140, may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Example depth sensors include active sensors, including one or more of indirect time-of-flight (iToF), direct time-of-flight (dToF), light detection and ranging (lidar), mmWave, radio detection and ranging (radar), and / or hybrid depth sensors (such as structured light). In embodiments without depth sensor 140, similar information regarding the depth of an object or a depth map can be generated passively from the disparity between the two image sensors (e.g., using parallax depth measurement or stereo depth measurement), a phase detection autofocus (PDAF) sensor, and the like. Furthermore, any number of additional image sensors or image signal processors may be present with device 100.

[0057] In some embodiments, image signal processor 112 may execute instructions from a memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in image signal processor 112, or instructions provided by processor 104. Additionally or alternatively, image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described herein. For example, image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), one or more automatic exposure compensation (AEC) 134 engines, and / or one or more video analytics engines (EVAs). AF 133, AEC 134, IFE 135, IPE 136, and EVA 137 may each include dedicated circuitry and may be embodied as software code executed by ISP 112, and / or a combination of hardware and software code executed on ISP 112.

[0058] In some implementations, memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described herein. In some implementations, instructions 108 include a camera application (or other suitable application) to be executed by device 100 for generating images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system and specific applications other than those for image or video generation. A camera application, such as that executed by processor 104, may enable device 100 to generate images using image sensors 101 and 102 and image signal processor 112. Memory 106 may also be accessible by image signal processor 112 to store processed frames or may be accessed by processor 104 to obtain processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be a circuit that includes image signal processor 112, and memory may be external to device 100. Device 100 may be coupled to external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112 , processor 104 , sensor hub 150 , memory 106 , and input / output components 116 into a single package.

[0059] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including the depth-of-focus blur operation. For example, execution of the instructions may instruct the image signal processor 112 to begin or end capturing an image frame or sequence of image frames, wherein the capture includes the depth-of-focus blur operation described in embodiments herein. In another example, execution of the instructions may instruct the image signal processor 112 to modify the captured image frame or sequence of image frames in a manner that includes the depth-of-focus blur operation. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs, such as the instructions 108 stored in the memory 106. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or video) stored in the memory 106.

[0060] When executing a camera application, processor 104 can be configured to instruct image signal processor 112 to perform one or more operations with reference to image sensor 101 or 102. For example, the camera application executing on processor 104 may receive a user command to initiate video preview display. Upon receiving this user command, video comprising a sequence of image frames from one or more image sensors 101 or 102 may be captured and processed by image signal processor 112. Image processing, such as that used to produce "output" or "corrected" image frames according to the techniques described herein, may be applied to one or more image frames in the sequence. Execution of instructions 108 by processor 104 outside of the camera application may also cause device 100 to perform any number of functions or operations. In some embodiments, processor 104 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124 or other coprocessor) to offload certain tasks from core 104A. AI engine 124 may be used to offload tasks associated with, for example, facial detection and / or object recognition, such as determining focus for applying a background blur effect. In some other embodiments, device 100 does not include processor 104 , such as when all described functionality is configured in image signal processor 112 .

[0061] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O components 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from a user and providing output to the user via display 114. For example, I / O components 116 may include (but are not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (such as a power button), a slider, a switch, etc.

[0062] Although shown as being coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, an application processor unit (APU), included in a system on a chip (SoC), or otherwise included in processor 104. Although reference is made to device 100 in the examples herein for performing aspects of the present disclosure, some device components may not be present. Figure 1100. In some embodiments, the present invention is not limited to a specific device or component configuration, including device 100.

[0063] Operable Figure 1 An exemplary image capture device is provided to obtain an improved image by processing an image based on an image captured by the image capture device to adjust a background blur effect of the image. Figure 2 One example method of operating one or more cameras, such as camera 103 , is shown in FIG. 1 and described below.

[0064] Figure 2 2 is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 of the system 200 can communicate with the image signal processor (ISP) 112 via a bidirectional bus and / or separate control and data lines. The processor 104 can control the camera 103 via a camera control 210, such as for configuring the camera 103 via a driver executing on the processor 104. The camera control 210 can be managed by a camera application 204 executing on the processor 104, which provides user-accessible settings so that the user can specify individual camera settings or select a profile with corresponding camera settings. The camera control 210 communicates with the camera 103 to configure the camera 103 according to commands received from the camera application 204. The camera application 204 can be, for example, a photography application, a document scanning application, a messaging application, or other application that processes image data acquired from the camera 103.

[0065] The camera configuration may include parameters specifying, for example, a frame rate, image resolution, readout duration, exposure level, aspect ratio, aperture size, etc. The camera 103 may obtain image data based on the camera configuration. For example, the processor 104 may execute the camera application 204 to instruct the camera 103 to set a first camera configuration of the camera 103 through the camera control 210, obtain first image data from the camera 103 operating in the first camera configuration, instruct the camera 103 to set a second camera configuration of the camera 103, and obtain second image data from the camera 103 operating in the second camera configuration.

[0066] In some embodiments where camera 103 is a variable aperture (VA) camera system, processor 104 may execute camera application 204 to instruct camera 103 to be configured to a first aperture size, obtain first image data from camera 103, instruct camera 103 to be configured to a second aperture size, and obtain second image data from camera 103. The reconfiguration of the aperture and the acquisition of the first and second image data may occur with little or no change in the scene captured at the first and second aperture sizes. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, and the like. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. That is, f / 2.0 is a larger aperture size than f / 8.0.

[0067] Image data received from camera 103 may be processed in one or more blocks of ISP 112 to form image frames 230 that are stored in memory 106 and / or provided to processor 104. In some embodiments, ISP 112 may apply effects to image frames 230. Processor 104 may further process the image data to apply the effects to image frames 230. Effects may include background blur, lighting, color casts, and / or high dynamic range (HDR) merging. In some embodiments, functionality may be embedded in different components, such as ISP 112, a DSP, an ASIC, or other custom logic circuitry for performing additional image processing.

[0068] Figure 2 The system 200 may be configured to perform reference Figure 3 The operations described are to determine the output image frame 230 . Figure 3 A flow chart illustrating an example method for processing image data to adjust a background blur effect of image data captured through a variable aperture lens, according to some embodiments of the present disclosure. Figure 3 Capturing in the image can result in an improved digital representation of the scene, resulting in a photo or video with higher image quality (IQ).

[0069] At block 302, first user input specifying a first focal depth is received from an input / output (I / O) component 116. In at least some embodiments, the first focal depth is one of a plurality of focal depths along a sliding scale. In other words, for example, the multiple depths of focus along the sliding scale may include all of the following values: f / 1.4, f / 1.5, f / 1.6, f / 1.7, f / 1.8, f / 1.9, f / 2.0, f / 2.1, f / 2.2, f / 2.3, f / 2.4, f / 2.5, f / 2.6, f / 2.7, f / 2.8, f / 2.9, f / 3.0, f / 3.1, f / 3.2, f / 3.3, f / 3.4, f / 3.5, f / 3.6, f / 3.7, f / 3.8, f / 3.9, and f / 4.0, but the camera is only able to capture images using the physical aperture at some of the aperture sizes on the sliding scale.

[0070] At block 304, the camera is configured with a first aperture size from a plurality of aperture sizes of the camera that is closest to a second aperture size corresponding to the first focal depth. The first aperture size is smaller than the second aperture size. The plurality of aperture sizes of the camera does not include the second aperture size. In other words, the camera may be a VA camera having available predetermined aperture sizes (e.g., f / 1.4, f / 2.0, f / 2.8, and f / 4.0), and the second aperture size corresponding to the specified focal depth (e.g., f / 3.2) is not included in the available predetermined aperture sizes.

[0071] At block 306, first image data representing a scene is received. For example, the first image data is received from an image sensor, such as when the image sensor is configured with a camera configuration. The first image data is captured using a first aperture size of the camera. The first image data may be received at ISP 112, processed by an image front end (IFE) and / or an image post-processing engine (IPE) of ISP 112, and stored in memory. In some embodiments, the capture of the image data may be initiated by a camera application executing on processor 104, which causes camera control 210 to activate the capture of image data by camera 103 and supply the image data to a processor, such as processor 104 or ISP 112.

[0072] At block 308, first modified image data is determined by applying a first blur kernel to the image frame. The first modified image data represents the scene at the depth of focus corresponding to the second aperture size. In other words, although the camera's VA does not actually include the second aperture size, the first modified image data simulates image data captured by the camera as if the VA did include the second aperture size.

[0073] The first blur kernel is determined based on a second aperture size corresponding to the depth of focus specified in user input. For example, a blur kernel lookup table may be generated using methods based on artificial intelligence (AI) or computer vision (CV). The first blur kernel is then determined from the blur kernel lookup table based on the second aperture size, the shape of the camera's aperture, and the focal distance to a target area (e.g., an object, subject, background, etc.) of the scene represented by the image data. This target area is selected by user input. The first blur kernel may be a Gaussian blur kernel, a Laplacian blur kernel, or the like. The user can select any area of ​​the scene as the focus target area, and thus, in at least some aspects, a blur kernel is generated for each instance of a focus area in the scene. Thus, when a specific area is selected as the target area, the blur kernel corresponding to that specific focus area is selected for application.

[0074] In at least some embodiments, a segmentation map is used as part of, or in addition to, applying a first blur kernel to an image frame. The segmentation map facilitates applying a desired degree of blur to each region of a scene represented in the image (e.g., an object, subject, background, etc.). In some aspects, the segmentation map can be obtained from a depth map via CV-based boundary extraction. In other aspects, the segmentation map can be obtained from a depth map via deep learning-based semantic segmentation. As used herein, a depth map is an image or image channel that contains information related to the distance of a surface of a scene region (e.g., an object) from the camera's viewpoint.

[0075] To help illustrate the effect of the first blur kernel, Figure 7 A depiction of a scene 700 represented by image data is shown. Scene 700 includes multiple regions (e.g., objects, subjects, background, portions of a scene, etc.), including person 702, person 704, a soccer ball 706, moon 708, and background 710. Person 702 and soccer ball 706 are located at a first, closest focal length from the camera, person 704 is located at a second, intermediate focal length from the camera, moon 708 is located at a third, intermediate focal length from the camera, and background 710 is located at a fourth, farthest focal length from the camera.

[0076] In this example, a VA camera includes apertures of f / 4.0, f / 2.8, f / 2.0, and f / 1.4. A user inputs a specified depth of focus, such that person 704 is in focus (is the target area) and corresponds to an aperture of f / 3.2 (e.g., a second aperture), which is between the camera's apertures of f / 4.0 and f / 2.8. The aperture of f / 4.0 is closest to, and smaller than, the specified aperture of f / 3.2, and therefore an image (e.g., image data) is captured using the aperture of f / 4.0 (e.g., the first aperture). A modified image (e.g., first modified image data) is determined by applying a first blur kernel to the captured image, which applies blur to the modified image to simulate an aperture of f / 3.2 in the image data captured at f / 4.0. In some instances, another area of ​​the scene (such as moon 708) may be in focus in the captured image, but is out of focus in the modified image after applying the first blur kernel. In some examples, another area of ​​the scene, such as background 710 , may be out of focus at a first degree of blurriness in the captured image and at a second, different degree of blurriness in the modified image after applying the first blur kernel.

[0077] return Figure 3 , first modified image data (e.g., image frame 230) may be determined by processor 104 or ISP 112 and stored in memory 106. The stored image frame may be read by processor 104 and used to form a preview display on a display of device 100 and / or processed to form a photo for storage in memory 106 and / or transmission to another device.

[0078] In various embodiments, the user may adjust the background blur effect of the captured image at a subsequent point in time. In such embodiments, method 300 may further include, after determining the first modified image data, receiving a second user input specifying a second focal depth different from the first focal depth. The second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size. The second modified image data may be determined by applying a second blur kernel to at least one of the first modified image data and the image data. The second modified image data represents the scene at the second focal depth. Continuing with Figure 7For example, user input specifying a depth of focus may later be received, such that person 702 and soccer ball 706 are in focus (are the target area) and correspond to an aperture size of f / 2.9 (e.g., a third aperture size), which is between the camera's aperture sizes of f / 4.0 and f / 2.8. A second modified image (e.g., second modified image data) is determined by applying a second blur kernel to the image captured using an aperture size of f / 4.0 (e.g., the first aperture size). This results in person 702 and soccer ball 706 being in focus in the second modified image. In another example, rather than person 702 and soccer ball 706 being in focus in the second modified image, person 704 is in focus in the second modified image, but with a modified blurriness of one or more areas of the scene.

[0079] Figure 4 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 or other processing circuitry may be configured to operate on image data to perform Figure 3 The user input and image data may be processed to determine one or more output image frames 410.

[0080] At block 404A, the processor 104 receives user input specifying a depth of focus. For example, the specified depth of focus corresponds to a focus target area of ​​a scene represented by the image data.

[0081] At block 404B, processor 104 determines a first aperture size from among a plurality of aperture sizes for the camera that is closest to a second aperture size corresponding to the first focal depth. The first aperture size is smaller than the second aperture size. For example, the camera may be a variable aperture (VA) camera having available predetermined aperture sizes (e.g., f / 1.4, f / 2.0, f / 2.8, and f / 4.0). A second aperture size corresponding to a specified focal depth (e.g., f / 3.2) is not included in the available predetermined aperture sizes.

[0082] At block 404C, the processor 104 receives image data. For example, such as when the image sensor is configured with a camera configuration, the processor 104 receives first image data from the image sensor. The first image data is captured using a first aperture size of the camera.

[0083] At block 404D, the processor 104 determines first modified image data by applying the first blur kernel to the image frame. The first modified image data represents image data at a depth of focus corresponding to the second aperture size. In other words, although the camera's VA does not actually include the second aperture size, the first modified image data simulates image data captured by the camera as if the VA did include the second aperture size.

[0084] Figure 2 The system 200 may be configured to perform reference Figure 5 The operations described are to determine the output image frame 230 . Figure 5 A flow chart illustrating an example method for processing image data to adjust a background blur effect for image data captured through an invariable (eg, fixed) aperture lens, according to some embodiments of the present disclosure. Figure 5 Capturing in the image can result in an improved digital representation of the scene, resulting in a photo or video with higher image quality (IQ).

[0085] At block 502, image data is received, the image data comprising a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus. For example, the first image data may be received from an image sensor, such as when the image sensor is configured with a camera configuration. The image data is captured by a camera with a first aperture size. In at least some examples, the camera is a fixed aperture camera with the first aperture size as the only available aperture size. The first image data may be received at ISP 112, processed by an image front end (IFE) and / or an image post-processing engine (IPE) of ISP 112, and stored in memory. In some embodiments, the capture of the image data may be initiated by a camera application executing on processor 104, which causes camera control 210 to activate the capture of image data by camera 103 and cause the image data to be provided to a processor, such as processor 104 or ISP 112.

[0086] At block 504, first user input specifying a first focal depth is received from input / output (I / O) component 116. The first focal depth corresponds to a second aperture size that is different from the first aperture size. In at least some instances, the first focal depth corresponds to an in-focus target area of ​​a scene represented by the image data.

[0087] At block 506, when the second aperture size is smaller than the first aperture size, a combined image frame is determined, the combined image frame comprising at least a first portion of the first image frame and a second portion of the second image frame. The first portion corresponds to the first focus point, and the second portion corresponds to the second focus point. The combined image frame is determined such that every region of the scene represented by the image data is in focus. In various embodiments, the first focus point is different from the second focus point, and the combined image frame is generated by fusing the first image frame with the second image frame. In a first example, the sharper regions from each of the first image frame having a near focus point and the second image frame having a far focus point are fused together to determine an image frame with full region focus. This first example may be used, for example, when the depth of focus of the captured scene is relatively deep. In another example, the focus may be placed on an intermediate region of the scene (e.g., not the closest or farthest region in the image), and a corresponding blur kernel may be applied to out-of-focus regions of the scene based on the defocus distance.

[0088] At block 508, first modified image data is determined from the combined image frame by applying a first blur kernel to the combined image frame when the second aperture size is less than the first aperture size. The first modified image data represents the scene at the depth of focus corresponding to the second aperture size. In other words, although the fixed aperture of the camera does not actually include the second aperture size, the second modified image data simulates image data captured by the camera as if the fixed aperture did include the second aperture size. The first blur kernel may be determined based on the second aperture size corresponding to the depth of focus specified in the user input, as described above with respect to Figure 4 described.

[0089] In various implementations, when the second aperture size is greater than the first aperture size, first modified image data is determined from the image data by applying a first blur kernel to the first image frame.

[0090] return Figure 7In this example, the fixed-aperture camera includes an aperture size of f / 2.0. User input specifies a depth of focus such that background 710 is in focus (is the target area) and corresponds to an aperture size of f / 4.0 (e.g., a second aperture size), which is smaller than the fixed-aperture camera's aperture size of f / 2.0. An image (e.g., image data) is captured by the fixed-aperture camera; however, when the image is captured using the aperture size of f / 2.0, background 710 is out of focus in the scene represented by the image. A fully focused image (e.g., first modified image data) is determined based on the image data. The fully focused image (e.g., second modified image data) is modified by applying a first blur kernel to the fully focused image, such that background 710 is in focus in the modified image. In some instances, another area of ​​the scene (such as moon 708) may be in focus in the captured image, but is out of focus in the modified image after application of the first blur kernel. In some examples, another area of ​​the scene, such as person 704 , may be out of focus at a first degree of blurriness in the captured image and at a second, different degree of blurriness in the modified image after applying the first blur kernel.

[0091] return Figure 5 , second modified image data (e.g., image frame 230) may be determined by processor 104 or ISP 112 and stored in memory 106. The stored image frame may be read by processor 104 and used to form a preview display on a display of device 100 and / or processed to form a photo for storage in memory 106 and / or transmission to another device.

[0092] In various embodiments, the user may adjust the background blur effect of the captured image at a subsequent point in time. In such embodiments, method 500 may further include, after determining the first modified image data, receiving a second user input specifying a second focal depth different from the first focal depth. The second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size. Second modified image data is determined by applying a second blur kernel to the image data. The second modified image data represents the scene at the second focal depth. Continuing with Figure 7For example, user input specifying a depth of focus may later be received such that moon 706 is in focus (is the target area) and corresponds to an aperture size of f / 2.4 (e.g., a third aperture size), which is smaller than the aperture size of f / 2.0 for a fixed aperture camera. A third modified image (e.g., second modified image data) is determined by applying a second blur kernel to the image captured using the aperture size of f / 2.0 (e.g., the first aperture size), resulting in moon 708 being in focus in the third modified image. In another example, rather than moon 708 being in focus in the third modified image, background 710 is in focus in the third modified image, but the blurriness of one or more areas of the scene is altered.

[0093] Figure 6 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 or other processing circuitry may be configured to operate on image data to perform Figure 5 The image data may be processed to determine one or more output image frames 610.

[0094] At block 604A, the processor 104 receives user input specifying a depth of focus. For example, the specified depth of focus corresponds to a focus target area of ​​the scene represented by the image data.

[0095] At block 604B, the processor 104 receives image data. For example, such as when the image sensor is configured with a camera configuration, the processor 104 receives image data from the image sensor. The image data is captured using a first aperture size of a camera (e.g., a fixed aperture camera).

[0096] At block 604C, the processor 104 determines first modified image data from the image data such that every region of the scene represented by the first modified image data is in focus in the first modified image data. In other words, the processor 104 determines a panoramically focused image from the image data.

[0097] At block 604D, the processor 104 determines second modified image data by applying the first blur kernel to the first modified image data. The second modified image data represents image data at a depth of focus corresponding to the second aperture size. In other words, although the fixed aperture of the camera does not actually include the second aperture size, the second modified image data simulates image data captured by the camera as if the fixed aperture did include the second aperture size.

[0098] One or more of the techniques described above for processing data from a VA camera system or a non-VA camera system to provide an adjustable background blur effect may be applied to an image capture device such as a Figure 8 As shown in . Figure 8 This block diagram illustrates an image processing technique for determining an image frame with an adjustable background blur effect, according to some embodiments of the present disclosure. A point spread function (PSF) convolution blur may be applied to either VA captured image data 820 or non-VA captured image data 810. VA captured image data 820 may be image data captured through a variable aperture lens at an aperture size corresponding to a user-selected depth of focus or aperture size. The aperture size may be small enough to obtain a representation of the scene with a sufficiently large blur depth to obtain a desired in-focus area, and blur is applied to the desired in-focus area at block 830 to achieve the effect of a larger aperture size in at least a portion of the image. Non-VA captured image data 810 may include image data captured from a lens with a fixed aperture size, which may be large, resulting in portions of the scene appearing blurred. The non-VA camera may be operated to obtain image data at different exposures at different depths of focus at block 812, and these different exposures may be combined at block 814 to obtain, at block 816, image frames with reduced blurriness, effectively obtaining a representation of the scene captured at a smaller aperture size.

[0099] At block 830, either the non-VA captured image data 810 or the VA captured image data 820 may be processed by applying the PSF at block 830 to increase the blurriness of a portion of the image. The blur may be applied via a segmentation map 850 input to block 830. The segmentation map 850 may provide information identifying segments of the scene. Each segment may have blur applied or not applied at block 830. In some embodiments, segments may receive different amounts of blur from the PSF at block 830. The PSF applied at block 830 may be determined by the optical blur block 840. The optical blur block 840 may use a depth map 842 to determine parameters for the PSF at block 844. For example, higher depth values ​​may result in a greater point spread. The PSF determined at block 844 may also be based on the focus target selected by the user at block 846. For example, the depth corresponding to the selected target may be used to apply a stronger blur from the PSF to segments that are farther from the selected target in depth. The PSF determined at block 844 may also be based on the aperture size selected by the user at block 848. For example, the aperture size may determine the blur strength, with a larger user-selected aperture size resulting in a blurrier or stronger PSF, and a smaller user-selected aperture size resulting in a less blurrier or weaker PSF.

[0100] The PSF determined at block 844 may be applied to the image data at block 830 to determine a corrected image frame that is a representation of the scene having an appearance corresponding to a user-selected aperture size or a user-selected depth of focus, but the camera does not have an available aperture size. Figure 8processing to provide editing of background blur effects in the representation of the scene.

[0101] according to Figure 8 Some aspects of image processing for capturing an image with background blur using a variable aperture (VA) camera are disclosed. When a user selects an aperture size for a photograph that is not in a VA configuration, a point spread function (PSF) kernel is interpolated from a smaller aperture and applied to a region of the image to achieve the desired blur. For example, if the true aperture size is f / 2.8 and the user selects an aperture size of f / 2.7, aspects of the processing described above can be used to apply true image blur to background regions of the image to enhance the background blur effect, corresponding to an aperture size of f / 2.7. While increasing blur is described in some embodiments, these techniques can also be applied to reduce blur to achieve a representation of a scene using an effective smaller aperture size.

[0102] In some embodiments, an image frame with a large depth of focus can be obtained by combining multiple exposures, resulting in an image frame with an effective small aperture size but a brightness similar to a large aperture size. Such an image frame can be referred to as a "full focus" frame. The background blur effect of the image frame can be adjusted by considering the distance and PSF on different sections of the image frame (such as by applying a different kernel from the blur kernel on the full focus image) to determine a corrected image frame.

[0103] In one or more aspects, techniques for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or apparatus described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to perform operations including: receiving a first user input specifying a first focal depth; configuring a camera to have a first aperture size among a plurality of aperture sizes of the camera, the first aperture size being closest to a second aperture size corresponding to the first focal depth, wherein the first aperture size is smaller than the second aperture size; receiving image data representing a scene captured using the first aperture size of the camera; and determining first modified image data by applying a first blur kernel to the image data. The first modified image data represents the scene at the focal depth corresponding to the second aperture size.

[0104] Additionally, the apparatus may perform or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, the apparatus includes a remote server (such as a cloud-based computing solution) that receives image data for processing to determine an output image frame. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein for the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, and the program code may be executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus. In some implementations, the apparatus may include one or more components configured to perform the operations described herein. In some implementations, the method of wireless communication may include one or more operations described herein with reference to the apparatus.

[0105] In a second aspect, in combination with the first aspect, the first focal depth is one of a plurality of focal depths along the sliding scale.

[0106] In a third aspect, in combination with one or more of the first or second aspects, the camera includes a variable aperture lens configured to capture images at each of a plurality of aperture sizes.

[0107] In a fourth aspect, in combination with the third aspect, the plurality of aperture sizes do not include the second aperture size.

[0108] In a fifth aspect, in combination with one or more of the first to fourth aspects, the first blur kernel is determined based on the second aperture size.

[0109] In a sixth aspect, in combination with one or more of the first to fifth aspects, the blur kernel reduces blurriness of at least a section of the representation of the scene.

[0110] In a seventh aspect, in combination with one or more of the first to sixth aspects, the blur kernel increases blurriness of at least a section of the representation of the scene.

[0111] In an eighth aspect, in combination with one or more of aspects 1 to 7, the operations further comprise: after determining the first modified image data, receiving a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; and determining second modified image data by applying a second blur kernel to at least one of the image data or the first modified image data, wherein the second modified image data represents the image data at the second focal depth.

[0112] In a ninth aspect, in combination with one or more of the first to eighth aspects, supporting image processing may include an apparatus configured to perform operations comprising: receiving image data comprising a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus, the image data being captured by a camera at a first aperture size; receiving a first user input specifying a first depth of focus corresponding to a second aperture size different from the first aperture size; and when the second aperture size is smaller than the first aperture size: determining a combined image frame comprising at least a first portion of the first image frame and a second portion of the second image frame, wherein the first portion corresponds to the first focus and the second portion corresponds to the second focus; and determining first modified image data from the combined image frame by applying a first blur kernel to the combined image frame, the first modified image data representing the scene at the depth of focus corresponding to the second aperture size.

[0113] In a tenth aspect, in combination with the ninth aspect, the first focal depth is one of a plurality of focal depths along the sliding scale.

[0114] In an eleventh aspect, in combination with one or more of the ninth to tenth aspects, the first aperture size of the camera is a fixed aperture.

[0115] In a twelfth aspect, in combination with one or more of the ninth to eleventh aspects, the first blur kernel is determined based on the second aperture size.

[0116] In a thirteenth aspect, in combination with one or more of aspects nine to twelve, the operations further comprise receiving a segmentation map corresponding to the image data, the segmentation map indicating a first segment and a second segment of a scene, and determining the first modified image data comprises applying the first blur kernel to the first segment of the combined image frame and applying the second blur kernel to the second segment of the combined image frame.

[0117] In a fourteenth aspect, in combination with one or more of the ninth to thirteenth aspects, the operations further comprise determining first modified image data from the image data by applying a first blur kernel to the first image frame when the second aperture size is greater than the first aperture size.

[0118] In a fifteenth aspect, in combination with one or more of aspects nine to fourteen, the operations further comprise: after determining the first modified image data, receiving a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; and determining second modified image data by applying a second blur kernel to the first modified image data, wherein the second modified image data represents the scene at the second focal depth.

[0119] It will be understood by those skilled in the art that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0120] This article is about Figures 1 to 2 、 Figure 4 and Figure 6 The components, functional blocks and modules described include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc., or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, processes, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages ​​or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0121] Those skilled in the art will understand that Figures 3 to 6 One or more blocks (or operations) described herein may be combined with one or more blocks (or operations) described with reference to another figure in each figure. Figure 4 One or more boxes (or operations) can be combined with Figures 1 to 3 For example, with one or more boxes (or operations) of Figure 6 One or more boxes associated with Figures 1 to 2 and Figure 5 A group of one or more boxes (or operations) that are associated.

[0122] It will also be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as resulting in departure from the scope of this disclosure. It will also be readily appreciated that the order or combination of components, methods, or interactions described herein are merely examples, and that the components, methods, or interactions of various aspects of this disclosure can be combined or performed in ways other than those illustrated and described herein.

[0123] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0124] The hardware and data processing apparatus for implementing the various illustrative logic components, logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuits specific to a given function.

[0125] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by, or for controlling the operation of, data processing apparatus.

[0126] If implemented in software, the functionality may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can be implemented to transfer a computer program from one location to another. Storage media can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. As used herein, magnetic disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where magnetic disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above are also intended to be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0127] Various modifications to the specific implementations described in this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein but are to be accorded the widest scope consistent with this disclosure, the principles and novel features disclosed herein.

[0128] Additionally, persons of ordinary skill in the art will readily recognize that, for convenience in describing the drawings, opposing terms such as "upper" and "lower" or "front" and "backward" or "top" and "bottom" or "forward" and "backward" are sometimes used and indicate relative positions corresponding to the orientation of the drawing on a correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0129] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be omitted from that combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0130] Similarly, although operations are depicted in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of flow charts. However, other operations not depicted may be combined in the schematically illustrated example processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any illustrated operations. In certain environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementations described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in different orders and still achieve the desired result.

[0131] As used herein, including in the claims, the term "or" used in a list of two or more items means that any one of the listed items can be employed alone, or any combination of two or more of the listed items can be employed. For example, if a composition is described as containing component A, B, or C, the composition can contain A alone; B alone; C alone; A and B combined; A and C combined; B and C combined; or A, B, and C combined. Furthermore, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one of" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C), or any combination of any of these.

[0132] The term "substantially" is defined as being largely, but not necessarily entirely, what is specified (and including what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed embodiment, the term "substantially" may be replaced with "within [percentage] of what is specified," where a percentage includes 0.1%, 1%, 5%, or 10%.

[0133] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method comprising: receiving a first user input specifying a first focal depth; configuring a camera to have a first aperture size of a plurality of aperture sizes of the camera, the first aperture size being closest to a second aperture size corresponding to the first depth of focus, wherein the first aperture size is smaller than the second aperture size; receiving image data comprising a representation of a scene captured using the first aperture size of the camera; as well as First modified image data is determined by applying a first blur kernel to the image data, wherein the first modified image data represents the scene at the depth of focus corresponding to the second aperture size. 2 . The method of claim 1 , wherein the first focal depth is one of a plurality of focal depths along a sliding scale. 3 . The method of claim 1 , wherein the camera comprises a variable aperture lens configured to capture images at each of the plurality of aperture sizes. The method of claim 3 , wherein the plurality of aperture sizes do not include the second aperture size. The method of claim 1 , wherein the first blur kernel is determined based on the second aperture size. The method of claim 1 , wherein the first blur kernel reduces blurriness of at least a segment of the representation of the scene. The method of claim 1 , wherein the blur kernel increases blurriness of at least a section of the representation of the scene.

8. The method according to claim 1, further comprising: receiving, after determining the first modified image data, a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; as well as Second modified image data is determined by applying a second blur kernel to at least one of the image data or the first modified image data, wherein the second modified image data represents the scene at the second depth of focus.

9. A device comprising: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations comprising: receiving a first user input specifying a first focal depth; configuring the camera to have a first aperture size of a plurality of aperture sizes of the camera, the first aperture size being closest to a second aperture size corresponding to the first depth of focus, wherein the first aperture size is smaller than the second aperture size; receiving image data representing a scene captured using the first aperture size of the camera; and First modified image data is determined by applying a first blur kernel to the image data, wherein the first modified image data represents the scene at the depth of focus corresponding to the second aperture size.

10. The apparatus of claim 9, wherein the first focal depth is one of a plurality of focal depths along a sliding scale.

11. The apparatus of claim 9, wherein the camera comprises a variable aperture lens configured to capture images at each of the plurality of aperture sizes.

12. The apparatus of claim 11, wherein the plurality of aperture sizes exclude the second aperture size.

13. The apparatus of claim 9, wherein the first blur kernel is determined based on the second aperture size. The device of claim 9 , wherein the first blur kernel reduces blurriness of at least a segment of the representation of the scene.

15. The device of claim 9, wherein the first blur kernel increases blurriness of at least a segment of the representation of the scene.

16. The apparatus of claim 9, wherein the operations further comprise: receiving, after determining the first modified image data, a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; as well as Second modified image data is determined by applying a second blur kernel to at least one of the image data or the first modified image data, wherein the second modified image data represents the image data at the second depth of focus.

17. A method comprising: receiving image data, the image data comprising a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus, the image data being captured by a camera at a first aperture size; receiving a first user input specifying a first depth of focus corresponding to a second aperture size different from the first aperture size; as well as When the second aperture size is smaller than the first aperture size: determining a combined image frame comprising at least a first portion of the first image frame and a second portion of the second image frame, wherein the first portion corresponds to the first focus and the second portion corresponds to the second focus; as well as First modified image data is determined from the combined image frame by applying a first blur kernel to the combined image frame, wherein the first modified image data represents the scene at the first depth of focus corresponding to the second aperture size.

18. The method of claim 17, wherein the first focal depth is one of a plurality of focal depths along a sliding scale. The method of claim 17 , wherein the first aperture size of the camera is a fixed aperture.

20. The method of claim 17, wherein the first blur kernel is determined based on the second aperture size.

21. The method according to claim 17, further comprising: receiving a segmentation map corresponding to the image data, the segmentation map indicating a first segment and a second segment of the scene, Wherein determining the first modified image data comprises applying the first blur kernel to a first section of the combined image frame and applying a second blur kernel to a second section of the combined image frame.

22. The method according to claim 17, further comprising: When the second aperture size is greater than the first aperture size: First modified image data is determined from the image data by applying a first blur kernel to the first image frame.

23. The method according to claim 17, further comprising: receiving, after determining the first modified image data, a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; as well as Second modified image data is determined by applying a second blur kernel to the first modified image data, wherein the second modified image data represents the scene at the second focal depth.

24. A device comprising: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations comprising: receiving image data, the image data comprising a first image frame representing a scene captured at a first focus and a second image frame captured at a second focus, the image data being captured by a camera at a first aperture size; receiving a first user input specifying a first depth of focus corresponding to a second aperture size different from the first aperture size; and When the second aperture size is smaller than the first aperture size: determining a combined image frame comprising at least a first portion of the first image frame and a second portion of the second image frame, wherein the first portion corresponds to the first focus and the second portion corresponds to the second focus; and First modified image data is determined from the combined image frame by applying a first blur kernel to the combined image frame, wherein the first modified image data represents the scene at the first depth of focus corresponding to the second aperture size.

25. The apparatus of claim 24, wherein the first focal depth is one of a plurality of focal depths along a sliding scale.

26. The apparatus of claim 24, wherein the first aperture size of the camera is a fixed aperture.

27. The apparatus of claim 24, wherein the first blur kernel is determined based on the second aperture size.

28. The apparatus of claim 24, wherein the operations further comprise: receiving a segmentation map corresponding to the image data, the segmentation map indicating a first segment and a second segment of the scene, Wherein determining the first modified image data comprises applying the first blur kernel to a first section of the combined image frame and applying a second blur kernel to a second section of the combined image frame.

29. The apparatus of claim 24, wherein the operations further comprise: When the second aperture size is greater than the first aperture size: First modified image data is determined from the image data by applying a first blur kernel to the first image frame.

30. The apparatus of claim 24, wherein the operations further comprise: receiving, after determining the first modified image data, a second user input specifying a second focal depth different from the first focal depth, wherein the second focal depth corresponds to a third aperture size different from the first aperture size and the second aperture size; as well as Second modified image data is determined by applying a second blur kernel to the first modified image data, wherein the second modified image data represents the scene at the second focal depth.

Images