Remove objects from an image

Abstract

Systems and techniques are provided for adjusting objects in an image. For example, the process may include acquiring a first image of a scene from a camera. The scene may include a first object located at a first location and a second object located at a second location. The process may include acquiring a second image of the scene from the camera. The second image of the scene may include the first object located at the first location and the second object located at a third location different from the second location. The process may include generating an adjusted second image based on the second image. The adjusted second image may include the first object located at the first location. The second object at the third location is removed from the adjusted second image. The process may include displaying the adjusted second image on a display.

Description

[Technical Field]

[0001] For example, aspects of the present disclosure relate to systems and techniques for removing objects from images. [Background technology]

[0002] Many devices and systems allow for capturing a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera, or a device that includes a camera, can capture a sequence of frames of a scene (e.g., a video of a scene). In some cases, the sequence of frames can be processed to perform one or more functions, output for display, or output for processing and / or consumption by other devices, among other uses.

[0003] When using a device to capture an image of a scene, there may be one or more objects in the scene that are not wanted in the final captured image. For example, a user may want to capture an image of a particular tourist in front of a national monument, and many passersby may be entering and exiting the scene.

[0004] A user can attempt to capture an image of a scene at the exact moment when there are no passersby in the scene, however, in some scenarios a scene without unwanted objects may not occur and / or a long waiting period may be required. Summary of the Invention

[0005] Systems, devices, methods, and computer-readable media for adjusting objects in an image are disclosed. According to at least one embodiment, a method for adjusting objects in an image is provided. The method includes: acquiring a first image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location; acquiring a second image of the scene from the camera, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; generating an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and displaying the adjusted second image on a display, the first image of the scene and the adjusted second image including images captured independent of obtaining a capture input.

[0006] In another embodiment, an apparatus for adjusting objects in an image is provided, the apparatus including at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: acquire a first image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location; acquire a second image of the scene from the camera, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; generate an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and display the adjusted second image on a display, the first image of the scene and the adjusted second image including images captured independent of obtaining a capture input.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: acquire from a camera a first image of a scene, the scene including a first object located at a first location and a second object located at a second location; acquire from the camera a second image of the scene, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; generate an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and display the adjusted second image on a display, the first image of the scene and the adjusted second image comprising images captured independent of obtaining a capture input.

[0008] In another embodiment, an apparatus for adjusting objects in an image is provided, the apparatus including: means for acquiring a first image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location; means for acquiring a second image of the scene from the camera, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; means for generating an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and means for displaying the adjusted second image on a display, the first image of the scene and the adjusted second image including images captured independent of obtaining a capture input.

[0009] In some aspects, one or more of the above-mentioned devices are, are part of, or include a mobile device (e.g., a mobile phone or so-called “smartphone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a vehicular computing device), or other device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors can be used to determine the location and / or orientation of the device, the state of the device, and / or for other purposes.

[0010] This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and claims.

[0011] The foregoing summary, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0012] With reference to the following figures, exemplary embodiments of the present application are described in detail below. [Brief explanation of the drawings]

[0013] [Figure 1A]1 is an image illustrating different fields of view within an example image, according to some embodiments of the present disclosure. [Figure 1B] 1 is an image illustrating different fields of view within an example image, according to some embodiments of the present disclosure. [Figure 1C] 1 is an image illustrating different fields of view within an example image, according to some embodiments of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating the architecture of an image capture and processing device, according to some embodiments of the present disclosure. [Figure 3] 1 is a block diagram illustrating an exemplary image adjustment system, according to some embodiments of the present disclosure. [Figure 4A] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 4B] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 4C] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 4D] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 4E] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 4F] 1 is an image illustrating an exemplary image adjustment according to some embodiments of the present disclosure. [Figure 5] 1 is a block diagram illustrating an example image capture system including an image adjustment system, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a flow diagram illustrating an exemplary image adjustment process, according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a block diagram illustrating an example of a deep learning network, in accordance with some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating an embodiment of a convolutional neural network, in accordance with some embodiments. [Figure 9]FIG. 1 illustrates one example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to one skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and description are not intended to be limiting.

[0015] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the present application, as set forth in the appended claims.

[0016] When capturing an image of a scene, unwanted objects (e.g., people, animals, vehicles, structures, etc.) may be present in the scene in addition to one or more target objects (e.g., people, animals, vehicles, etc.). FIGS. 1A-1C are images illustrating a scene including unwanted objects and a process for removing the unwanted objects from an image of the scene. FIG. 1A is an image 100 of two people 102 sitting on a staircase. In image 100, person 102 is a target object (e.g., subject) desired to be captured in the scene of image 100. In FIG. 1A, image 100 also includes three people 104A, 104B, and 104C located behind person 102. In some cases, it may be desirable to remove people 104A, 104B, and 104C from the background of image 100.

[0017] FIG. 1B shows a segmented image 110 in which the three people 104A, 104B, and 104C from FIG. 1A are indicated by segmentation contours 106A, 106B, and 106C. For example, the three people 104A, 104B, and 104C may be selected and identified (e.g., by user input, by a machine learning model, etc.) for removal from the segmented image 110. In some cases, a segmentation process may be used to determine which pixels in the segmented image 110 belong to each of the people 104A, 104B, and 104C, as indicated by the segmentation contours 106A, 106B, and 106C. In some cases, the segmentation shown in the segmented image 110 may be performed, in whole or in part, by a machine learning model. For example, the machine learning model may be trained to detect features within the segmented image 110. In some cases, based on the detected features within the segmented image 110, the machine learning model may classify objects within the segmented image 110. Exemplary classifications for people 104A, 104B, 104C may include "people," "background objects," etc. In some cases, the user may be able to select which objects (or classifications) within the segmented image 110 to target for adjustment (e.g., zooming in).

[0018] In some aspects, adjusting image 100 by removing people 104A, 104B, and 104C may leave blank areas in the resulting adjusted image. FIG. 1C shows adjusted image 120 in which the blank areas left by the removal of people 104A, 104B, and 104C in region 108 have been filled. In the illustrated embodiment, filled region 108 appears to include sky and clouds that match the background. In some cases, these portions of adjusted image 120 can be filled using an inpainting process. For example, the inpainting process can attempt to fill the blank areas of adjusted image 120 with colors that approximate (or estimate) the portions of the scene captured in image 100 that were obscured by people 104A, 104B, and 104C. In one example embodiment, inpainting can be performed by interpolating and / or blending colors from pixels in input image 100 adjacent to the blank areas and applying the interpolated colors to the blank areas. In some cases, the contours of removed objects (e.g., people 104A, 104B, 104C) may remain visible after replacing the blank areas with estimated color values. In some cases, image adjustment techniques (e.g., inpainting processes) may perform blending and smoothing around the edges of people 104A, 104B, 104C (e.g., along segmented contours 106A, 106B, 106C).

[0019] In another illustrative example, one or more additional images (not shown) of the same scene or portion of the same scene as segmented image 110 may include portions of the scene that are occluded by the people 104A, 104B, 104C in segmented image 110. In some examples, blank areas created by removing unwanted objects 104A, 104B, 104C in segmented image 110 may be filled by combining pixels from the one or more additional images with image 100, adjusted image 120, one or more additional images, or any combination thereof. However, in some cases, one or more additional images of a scene or portion of a scene may not have been captured and / or stored. As a result, techniques for combining pixels from multiple images may not be available.

[0020] What is needed are systems and techniques for identifying objects for removal from an image during the image capture process and removing the identified objects. For example, before capturing an image of a scene (e.g., before a capture input is received), unwanted objects appearing in a preview image (e.g., displayed on a display) can be identified for removal. As a result, the preview image can be stored in memory and match the final captured image, similarly having the unwanted objects removed.

[0021] Described herein are systems, devices, processes (also referred to as methods), and computer-readable media (collectively "systems and techniques") for identifying objects for removal from an image during an image capture process and removing the identified objects. In some cases, after one or more unwanted objects are identified for removal, an additional preview image may be adjusted to remove the unwanted objects before capture. In some embodiments, when an image of a scene is captured (e.g., in response to a capture input), the unwanted objects may be removed from the scene before storing the adjusted image in storage. For example, a photographer may be able to view a preview image showing the unwanted objects removed before pressing the shutter (or providing any other type of capture input).

[0022] Various aspects of the techniques described herein are discussed below with reference to the figures. FIG. 2 is a block diagram illustrating the architecture of an image capture and processing system 200. The image capture and processing system 200 includes various components used to capture and process an image of a scene (e.g., an image of scene 210). The image capture and processing system 200 can capture a single image (or photograph) and / or can capture a video including multiple images (or video frames) in a particular sequence. A lens 215 of the image capture and processing system 200 is aimed at the scene 210 and receives light from the scene 210. The lens 215 bends the light toward the image sensor 230. The light received by the lens 215 passes through an aperture controlled by one or more control mechanisms 220 and is received by the image sensor 230.

[0023] The one or more controls 220 can control exposure, focus, and / or zoom based on information from the image sensor 230 and / or based on information from the image processor 250. The one or more controls 220 can include multiple mechanisms and components, for example, the control 220 can include one or more exposure controls 225A, one or more focus controls 225B, and / or one or more zoom controls 225C. The one or more controls 220 can also include additional controls beyond those shown, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0024] The focus control mechanism 225B of the control mechanism 220 can obtain the focus setting. In some embodiments, the focus control mechanism 225B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 225B can adjust the position of the lens 215 relative to the position of the image sensor 230. For example, based on the focus setting, the focus control mechanism 225B can adjust the focus by actuating a motor or servo (or other lens mechanism) to move the lens 215 closer to or farther from the image sensor 230. In some cases, additional lenses, such as one or more microlenses over each photodiode of the image sensor 230, can be included in the image capture and processing system 200, each of which bends light received from the lens 215 toward a corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 220, the image sensor 230, and / or the image processor 250. The focus setting may also be referred to as the image capture setting and / or the image processing setting.

[0025] An exposure control 225A of the control mechanism 220 can obtain an exposure setting. In some cases, the exposure control 225A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control 225A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 230 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 230, or any combination thereof. The exposure setting may also be referred to as an image capture setting and / or an image processing setting.

[0026] The zoom control 225C of the control mechanism 220 can obtain a zoom setting. In some embodiments, the zoom control 225C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control 225C can control the focal length of an assembly of lens elements (lens assembly) that includes the lens 215 and one or more additional lenses. For example, the zoom control 225C can control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more of the lenses relative to one another. The zoom setting may also be referred to as an image capture setting and / or an image processing setting. In some embodiments, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some embodiments, the lens assembly may include a focusing lens (which in some cases may be lens 215) that first receives light from scene 210; the light then passes through an afocal zoom system between the focusing lens (e.g., lens 215) and image sensor 230 before it reaches image sensor 230. The afocal zoom system may include two positive (e.g., convergent, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference of each other), in some cases with a negative (e.g., divergent, concave) lens between them. In some cases, zoom control 225C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0027] Image sensor 230 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image generated by image sensor 230. In some cases, different photodiodes can be covered by different color filters, thus measuring light that matches the color of the filter covering that photodiode. For example, a Bayer color filter includes red, blue, and green filters, and each pixel of the image is generated based on red light data from at least one photodiode covered in the red filter, blue light data from at least one photodiode covered in the blue filter, and green light data from at least one photodiode covered in the green filter. Other types of color filters can use yellow, magenta, and / or cyan (also referred to as "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors (e.g., image sensor 230) may lack color filters altogether and instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes across the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color depth due to the lack of color filters.

[0028] In some cases, image sensor 230 may alternatively or additionally include opaque and / or reflective masks that can be used for phase-detection autofocus (PDAF) to block light from reaching particular photodiodes, or portions of particular photodiodes, at particular times and / or from particular angles. Image sensor 230 may also include analog gain amplifiers for amplifying analog signals output by the photodiodes and / or analog-to-digital converters (ADCs) for converting analog signals output from the photodiodes (and / or amplified by the analog gain amplifiers) to digital signals. In some cases, particular components or functions discussed with respect to one or more of control mechanisms 220 may alternatively or additionally be included within image sensor 230. The image sensor 230 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complimentary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0029] Image processor 250 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 254), one or more host processors (including host processor 252), and / or one or more of any other types of processors 910 discussed with respect to computing system 900. Host processor 252 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 250 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 252 and ISP 254. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 256), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components.I / O ports 256 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one exemplary embodiment, host processor 252 may communicate with image sensor 230 using an I2C port, and ISP 254 may communicate with image sensor 230 using a MIPI port.

[0030] Image processor 250 may perform any number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging images to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 250 may store image frames and / or processed images in random access memory (RAM) 140 / 3225, read-only memory (ROM) 145 / 920, a cache, a memory unit, another storage device, or some combination thereof.

[0031] Various input / output (I / O) devices 260 may be connected to image processor 250. I / O devices 260 may include a display screen, a keyboard, a keypad, a touch screen, a trackpad, a touch-sensitive surface, a printer, any other output device(s) 935, any other input device(s) 945, or some combination thereof. In some cases, captions may be entered into image processing device 205B via a physical keyboard or keypad of I / O device 260 or via a virtual keyboard or keypad of a touch screen of I / O device 260. I / O 260 may include one or more ports, jacks, or other connectors that enable a wired connection between image capture and processing system 200 and one or more peripheral devices, through which image capture and processing system 200 may receive data from and / or send data to one or more peripheral devices. I / O 260 may include one or more wireless transceivers that enable a wireless connection between image capture and processing system 200 and one or more peripheral devices, through which image capture and processing system 200 can receive data from and / or transmit data to one or more peripheral devices. Peripheral devices may include any of the aforementioned types of I / O devices 260 and may themselves be considered I / O devices 260 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0032] In some cases, image capture and processing system 200 can be a single device. In some cases, image capture and processing system 200 can be two or more separate devices including image capture device 205A (e.g., a camera) and image processing device 205B (e.g., a computing device coupled to the camera). In some implementations, image capture device 205A and image processing device 205B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, image capture device 205A and image processing device 205B can be disconnected from each other.

[0033] 2 into two portions, representing image capture device 205A and image processing device 205B, respectively. Image capture device 205A includes lens 215, control mechanism 220, and image sensor 230. Image processing device 205B includes image processor 250 (including ISP 254 and host processor 252), RAM 240, ROM 245, and I / O 260. In some cases, certain components shown in image capture device 205A, such as ISP 254 and / or host processor 252, may also be included within image capture device 205A.

[0034] Image capture and processing system 200 may include an electronic device such as a mobile or fixed telephone handset (e.g., a smartphone, a mobile phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some embodiments, image capture and processing system 200 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, image capture device 205A and image processing device 205B may be different devices. For example, image capture device 205A may include a camera device, and image processing device 205B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0035] While image capture and processing system 200 is shown to include certain components, those skilled in the art will understand that image capture and processing system 200 may include many more components than those shown in FIG. 2 . The components of image capture and processing system 200 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 200 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 200. In some embodiments, image adjustment system 300 may include image capture and processing system 200, image capture device 205A, image processing device 205B, or a combination thereof.

[0036] 3 is a block diagram illustrating one embodiment of an image adjustment system 300. The image adjustment system 300 includes various components used to process one or more images, such as removing unwanted features or objects in the one or more images. The image adjustment system 300 can remove the unwanted features or objects and replace the color values of pixels at the locations of the removed objects with colors that represent portions of the scene captured in the one or more images that are occluded by the removed objects. As shown, the components of the image adjustment system 300 include one or more image capture devices 302, an object identification engine 304, a feature segmentation engine 306, and an image adjustment engine 308.

[0037] 3 and the following description of image adjustment system 300, various aspects of image adjustment performed by image adjustment system 300 are illustrated with reference to the exemplary images shown in FIGS. 4A-4F. In the exemplary illustrations of FIGS. 4A-4F, a person 402 is selected for adjustment (e.g., removal and / or replacement). In the illustrated embodiment, the adjustment applied to person 402 in FIG. 4A is to remove the person and replace them with background objects 404 that are occluded by person 402 from any images captured after person 402 is identified as an unwanted object.

[0038] 3, image adjustment system 300 may include or be part of a mobile or fixed telephone handset (e.g., a smartphone, a cellular phone, etc.), a server computer (e.g., in communication with a vehicle computing system), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some embodiments, image adjustment system 300 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications, such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications. In some implementations, the components of the image adjustment system 300 (e.g., one or more image capture devices 302, the object identification engine, the feature segmentation engine 306, and the image adjustment engine 308) may be part of the same computing device. In some implementations, the components of the image adjustment system 300 may be part of two or more separate computing devices. In some cases, the image adjustment system 300 may be implemented as part of a computing system 900 shown in FIG. 9.

[0039] While image adjustment system 300 is shown to include certain components, one skilled in the art will understand that image adjustment system 300 may include more or fewer components than those shown in FIG. 3. In some cases, the additional components of image adjustment system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, image adjustment system 300 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 3. In some implementations, additional components of image adjustment system 300 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, for performing various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image adjustment system 300.

[0040] One or more image capture devices 302 can capture one or more images. The one or more image capture devices 302 (e.g., cameras or other image sensors) can be included within a mobile device and can be oriented toward a user of the device (e.g., using one or more front-facing cameras) or away from a user of the device (e.g., using one or more rear-facing cameras).

[0041] Each of the one or more image capture devices 302 may include a camera or other type of image sensor. In some embodiments, the one or more image capture devices 302 may include an IR camera configured to capture infrared (IR) images and / or near-infrared (NIR) images. For example, the IR camera or IR sensor may capture IR signals. IR signals have wavelengths and frequencies within the IR electromagnetic spectrum. The IR electromagnetic spectrum includes wavelengths ranging from 2500 nanometers (nm) to 1 millimeter (mm), corresponding to frequencies ranging from 430 terahertz (THz) to 400 gigahertz (GHz). The infrared spectrum includes the NIR spectrum, which includes wavelengths ranging from 780 nm to 2500 nm. In some cases, the image adjustment system 300 may include an IR sensor configured to capture IR and NIR signals. In some cases, separate IR and NIR sensors may be included in image adjustment system 300. In some embodiments, one or more image capture devices 302 may include cameras configured to capture color and / or monochrome images. Color images may include red-green-blue (RGB) images, luminance, blue-difference, red-difference (YCbCr or Y′CbCr) images, and / or any other suitable type of image. In an exemplary embodiment, image adjustment system 300 may include one or more RGB cameras. In some cases, one or more image capture devices 302 may include one or more IR cameras and one or more RGB cameras.

[0042] In some embodiments, the one or more image capture devices 302 may include one or more depth sensors. The one or more depth sensors may obtain distance measurements corresponding to objects within the captured scene. In one illustrative example, the depth sensor may take the form of a light source capable of projecting a structured or textured light pattern, which may include one or more narrowband light, onto one or more objects within the scene. Depth information may then be obtained by exploiting geometric distortions of the projected pattern caused by the surface shape of the object. In some cases, the one or more depth sensors may generate a depth image including depth values corresponding to pixel locations within one or more images captured by the one or more image capture devices 302. In some cases, the depth sensor may be located in the same general location as other sensors of the one or more image capture devices 302. In some cases, the depth sensor may capture a depth image simultaneously with images captured by one or more other sensors included within the one or more image capture devices 302. In some implementations, the systems and techniques described herein can also be used when depth information is inferred from one or more images.

[0043] In one exemplary embodiment, the one or more image capture devices 302, and in some cases the one or more depth sensors, can capture one or more RGB-D images. In some cases, the one or more image capture devices 302 can capture other image types that include depth information, such as monochrome depth, NIR depth, etc. For illustrative purposes, the embodiments of this disclosure discuss performing image adjustments on RGB-D images, but the systems and techniques described herein can also be used with other image types that include depth information without departing from the scope of this disclosure.

[0044] 4A shows an exemplary first input image 410 of a scene. In the illustrated example, the first input image 410 includes a person 402 positioned in front of background objects 404. For example, the background objects 404 include trees and several tents. For purposes of this example, an image adjustment process will be described to remove the person 402 from the image of the scene while maintaining the accurate appearance of the background objects 404.

[0045] Returning to FIG. 3 , one or more images captured by one or more image capture devices 302 may be provided as input to object identification engine 304. In one exemplary embodiment, first input image 410 of FIG. 4A may be the first input image captured by the image capture device and displayed on a display. In some cases, the first input image may be displayed to provide a user with a preview of the image that will be captured after the image capture device receives the capture input (e.g., when the user presses a capture button). In the embodiment of FIG. 4B , identification information 406 corresponding to the location of person 402 of FIG. 4A may be obtained by object identification engine 304. For example, object identification engine 304 may obtain an indication of one or more selected objects (or ROIs) within the input image to be adjusted. In some cases, the indication of one or more selected objects within the input image to be adjusted may be based on user input. In one exemplary implementation, a user may be able to indicate one or more selected objects using physical contact (e.g., tapping, swiping, etc.) on the screen, a gesture (e.g., detected by a camera or other sensor), using an input device, or through any other means that allows the user to interact with the image adjustment system 300.

[0046] In some cases, pixel locations of identified objects may be provided as input to the feature segmentation engine 306. For illustrative purposes, a feature segmentation engine 306 based on a machine learning model (e.g., a deep learning neural network) is described below. However, it should be understood that feature segmentation based on other techniques, such as computer vision, digital image processing, thresholding, region-based segmentation, edge segmentation, Otsu's algorithm, clustering algorithms, any other feature segmentation technique, or any combination thereof, may be used without departing from the scope of this disclosure. In some cases, the feature segmentation engine 306 may be trained to detect spatial information (e.g., features) associated with one or more images. In some cases, the feature segmentation engine 306 may be further trained to provide one or more classifications for objects in one or more images based on the detected features. The feature segmentation engine 306 may then use those classifications to segment the image into various portions associated with one or more classifications. For example, the feature segmentation engine 306 may segment one or more images into various parts associated with people, buildings, cars, furniture, plants, and the like.

[0047] The feature segmentation engine 306 can also be trained to classify objects corresponding to features extracted from one or more images using one or more classifications. A training data set including example images and classification labels can be used to train the feature segmentation engine 306 using techniques such as those described with respect to FIGS. 7 and 8. In one example implementation, during inference (e.g., after the feature segmentation engine 306 is trained), the feature segmentation engine 306 can classify one or more input images using the feature vectors. Example classifications may include "person," "face," "building," "tree," and any other classification that the feature segmentation engine 306 is trained to classify.

[0048] In one exemplary implementation, the feature segmentation engine 306 may perform semantic segmentation of the image. In semantic segmentation, the feature segmentation engine 306 may associate each object in a scene with one or more classifications (also referred to herein as labels). In some cases, if two or more objects have the same label, semantic segmentation does not distinguish between the two objects. In such implementations, depth information in one or more captured images (e.g., RGB-D images) may be used to distinguish and further segment the objects. For example, a portion of an input image containing two people may be classified with a single classifier, “people.” FIG. 4B illustrates an exemplary image 420 showing the classifier 408 (represented as white) associated with the person 402 in the first input image 410. In some implementations, the classifier 408 may represent a classification such as “people” or “face.” In some aspects, with semantic segmentation, the feature segmentation engine 306 may not be able to separately identify overlapping objects as different objects.

[0049] In another example implementation, the feature segmentation engine 306 can perform instance segmentation. In instance segmentation, the feature segmentation engine 306 can separately identify multiple object instances with the same classifier. For example, if multiple people are present in a single image, such as image 100, instance segmentation can be used to distinguish five individual people (e.g., two people 102 and people 104A, 104B, and 104C) by assigning a separate classifier to each identified person. As a result, a subset of people (e.g., people 104A, 104B, and 104C) can be selected for removal, while a different subset of people (e.g., person 102) can be retained in the final adjusted image.

[0050] In some cases, based on the features, objects, and / or classifications determined from the first input image, the object identification engine 304, the feature segmentation engine 306, and / or the image adjustment engine 308 may associate pixel locations in an input image (e.g., the first input image 410 of FIG. 4A , the second input image 440 of FIG. 4D ) with the corresponding features, objects, and / or classifications determined by the feature segmentation engine 306 from that input image. In some cases, the pixel locations associated with the unwanted objects may be provided as input to the image adjustment engine 308. In some cases, the features, objects, and / or classifications associated with the unwanted objects may be stored in a storage device (not shown) and used to identify the unwanted objects in subsequent images.

[0051] Once the image adjustment engine 308 obtains pixel locations associated with the unwanted objects from the object identification engine 304 and / or feature segmentation engine 306, the image adjustment engine 308 can adjust those pixel locations associated with the unwanted objects in each image. In some cases, based on the identified pixel locations, the image adjustment engine 308 can form an adjusted image by removing and / or replacing the unwanted objects.

[0052] In some cases, the image adjustment engine 308 may perform an inpainting process to replace pixels previously occupied by the unwanted object in the input image. In some cases, the image adjustment engine 308 may acquire one or more additional images of the same scene or portion of a scene as the first input image. In some cases, pixels obscured by an unwanted object (e.g., person 402) in the first input image may not be obscured in another image of the scene or portion of the scene processed by the image adjustment system 300. In some cases, unobscured pixel information corresponding to the location of the unwanted object in the first input image may be used to replace the unwanted object in the first adjusted image. For example, the adjusted image may be generated by combining pixels from two or more images (e.g., image fusion). FIG. 4C shows a first adjusted image 430 in which the unwanted object (e.g., person 402) has been removed and replaced (e.g., at location 412 in FIG. 4C ) by the image adjustment engine 308 of the image adjustment system 300.

[0053] In some cases, once an unwanted object has been identified, classified, and / or removed by the image adjustment system 300, the features, objects, and / or classifications associated with the unwanted object may be retrieved from storage (not shown) and used to identify unwanted objects in subsequent images.

[0054] Figure 4D shows a second input image 440 that also includes the same person 402, but in a different location within the same scene as captured in Figure 4A. In some cases, the feature segmentation engine 306 can determine a classifier 408 for the person 402 from the second image 440. Accordingly, the image 450 shown in Figure 4E shows the identity 416 of the person 402 in a new location within the second input image. In some cases, the object identification engine 304 can take the classification generated by the feature segmentation engine 306 and determine that the identified object matches a common object present in both the first input image 410 and the second input image 440. Based on the object identification engine 304's identification of the unwanted object in the second input image 440, the image adjustment engine 308 can remove and / or replace the person 402 from the background objects 404 to generate a second adjusted image 460, as shown in Figure 4F. 4F, the replaced pixels 418 may correspond to unobscured pixels from corresponding locations in the first input image 410 and / or the first adjusted image 430, as indicated by the white outlines 414. As additional images are acquired from the one or more image capture devices 302, the image adjustment system 300 may track identified unwanted objects in a similar manner and remove and / or replace the unwanted objects.

[0055] While the embodiments described herein are provided in terms of identifying and removing unwanted objects, in some cases, target objects desired for inclusion in the adjusted image can be identified, and the image adjustment system 300 can determine whether any objects not identified as target objects are present in the scene. For example, the object identification engine 304 and / or the feature segmentation engine 306 can determine whether an object in the scene that was not identified as a target object has moved, and identify the moved object as an object for removal. In another exemplary technique, depth information captured by one or more image capture devices 302 can be used to determine unwanted objects. For example, if the depth of the target object exceeds the depth of another detected object (e.g., detected by the object identification engine 304) by a threshold amount, the other detected object can be identified as an unwanted object. Regardless of the technique used to identify the target object and / or the unwanted object, the systems and techniques described herein can be used to remove and / or replace the unwanted objects from the adjusted image.

[0056] In some cases, the object identification engine 304 can receive additional input to guide the identification of objects as target objects and / or unwanted objects. For example, the object identification engine 304 can receive user input and / or voice input to guide the object identification. In one exemplary embodiment, the user input may include recognizing only family members, friends, and / or pets. For example, the object identification engine 304 can be trained to recognize people, places, pets, etc. from a photo album stored on a storage device (not shown). In another exemplary embodiment, the object identification engine 304 can receive user gaze information to determine a region of interest corresponding to the user's gaze. In some implementations, the object identification engine 304 can determine that objects outside the region of interest are unwanted objects. In some cases, the object identification engine 304 can identify target objects within the region of interest. In another exemplary embodiment, the object identification engine 304 can receive voice input corresponding to the input image to determine the target objects and / or unwanted objects. For example, the object identification engine 304 may determine that a captured sound corresponds to a family member, friend, pet, etc., and determine an area of interest within the captured image that corresponds to the source of the captured sound. In another exemplary embodiment, the object identification engine 304 may determine that a captured sound corresponds to an unrecognized person or pet, and identify one or more unwanted objects within the captured image that correspond to the source of the captured sound.

[0057] In the illustrated embodiment, FIG. 3 shows the object identification engine 304 separate from the feature segmentation engine 306; in some cases, identifying one or more objects for removal and / or one or more target objects can be performed after the input image is segmented by the feature segmentation engine 306. For example, the feature segmentation engine 306 can be trained to identify people, places, pets, etc. as target objects in the input image. For example, the feature segmentation engine 306 can be trained to recognize features in the input image that correspond to previously photographed people, places, pets, etc. In some cases, the object identification engine can take as input the features determined by the feature segmentation engine 306 and can determine unwanted objects in the scene captured in the input image based on other factors, such as the movement of the objects between subsequent images.

[0058] In some cases, the image adjustment system 300 can acquire multiple images of a scene or portion of a scene captured in an input image (e.g., first input image 410 of FIG. 4A , second input image 440 of FIG. 4D ). For example, the electronic device can include a first image capture device and a second image capture device (e.g., included in one or more image capture devices 302) that simultaneously capture at least a partially common portion of a scene. In some cases, the object identification engine 304 can determine a target object and / or unwanted objects based on the image acquired from the first image capture device and remove and / or replace the unwanted objects from the image acquired from the second image capture device. In an exemplary embodiment, the first image capture device can have a lower resolution and / or lower power consumption than the second image capture device. For example, the first image capture device can include an always-on (AON) camera. In some cases, a single image sensor can capture multiple images of a scene to provide, for example, zero shutter lag (ZSL) capture, preview images, etc.

[0059] FIG. 5 is a block diagram illustrating an exemplary image capture system 500. As shown, image capture system 500 includes a ZSL buffer 532, an image processing engine 540, an image adjustment engine 545, and a preview buffer 536. Image adjustment engine 545 may correspond to image adjustment system 300 shown in FIG. 3. It should be understood that image capture system 500 is an exemplary block diagram provided for illustrative purposes and may include more or fewer components without departing from the scope of the present disclosure. Furthermore, while image adjustment engine 545 is shown as being included within image processing engine 540, image adjustment engine 545 may be separate from image processing engine 540 without departing from the scope of the present disclosure.

[0060] As shown in FIG. 5 , a timeline 501 for an exemplary image capture sequence includes a capture event 503 (e.g., a capture input). In the illustrated embodiment, images from an image sensor 530 (e.g., image sensor 230 of FIG. 2 , one or more image capture devices 302 of FIG. 3 , etc.) may be captured during a preview and ZSL period 502 before the capture event 503. Additionally, an image capture period 504 after the capture event 503 may include capturing images after the capture event 503. In the exemplary embodiment of FIG. 5 , the image capture system 500 may be configured to provide a preview image with unwanted objects in the preview image removed (e.g., before receiving the capture input). In some cases, the image capture system 500 may provide a preview image before, during, and / or after the capture event 503.

[0061] In some implementations, the image capture system 500 can acquire images from an image sensor 530, which may correspond to one or more image capture devices 302 of FIG. 3. As shown, a zero shutter lag (ZSL) buffer 532 can be used to store images or frames captured by the image sensor 530. In some embodiments, the ZSL buffer 532 is a circular buffer. In general, the ZSL buffer 532 can be used to store one or more frames recently captured by the sensor, thereby compensating for any delay that may occur before the image capture system 500 finishes encoding and storing a frame in response to a shutter (or capture) command being received (e.g., based on user input or automatically). The ZSL buffer 532 can be coupled to an image processing engine 540 to perform image processing on the image before it is output and stored in a storage device (e.g., by the storage device 930 of FIG. 9). For example, the image processing engine 540 can perform image processing on the image in response to a capture event 503.

[0062] In some cases, the image processing engine 540 may generate a preview image and output the preview image to the preview buffer 536. For example, the image processing engine 540 may generate the preview image by downscaling and / or cropping raw image data from the image sensor 530 to match the size of a preview display (e.g., the display of an electronic device). In some cases, as described above, the image adjustment engine 545 may identify unwanted objects in the preview image (e.g., based on user input or determined automatically). In some cases, once the unwanted objects are identified, the image adjustment engine 545 may remove and / or replace the unwanted objects from the preview image and provide the adjusted preview image to the preview buffer 536. As a result, the image output to the preview display 550 may provide an indication of what the captured image would look like with the unwanted objects removed. In some cases, the image adjustment engine 545 may utilize an inpainting process to replace pixels associated with the unwanted objects during the preview and ZSL period 502. For example, an inpainting process using interpolation may be used to avoid delays associated with combining pixels from multiple images. In some cases, segmentation (e.g., by feature segmentation engine 306 of FIG. 3) can be performed on a downscaled and / or cropped preview image to determine the location of unwanted objects. In some cases, each preview image stored in preview buffer 536 can be indexed to a corresponding image stored in ZSL buffer 532. In some cases, the segmentation provided by feature segmentation engine 306 can be stored in preview buffer 536 in addition to or instead of the preview image.

[0063] In some cases, after a capture event 503 occurs, the image processing engine 540 may obtain an image (referred to herein as a processed image) from the ZSL buffer 532, process it, and output it to storage. In some cases, this processing of the processed image may be performed during the image processing period 506. As described above, each image in the ZSL buffer 532 may be indexed to a corresponding preview image and / or segmentation in the preview buffer 536. In some cases, the segmentation determined for the preview image may be reused by the image adjustment engine 545 to identify pixels in the processed image that correspond to unwanted objects. In some cases, if an unwanted object is located in a different position in the scene in the images stored in the ZSL buffer 532, pixels obscured by the unwanted object in the processed image may be replaced with corresponding pixels (e.g., at the obscured pixel location) from other images in the ZSL buffer 532 that were not obscured by the unwanted object. However, in some cases, none of the images stored in the ZSL buffer 532 may include an unobstructed view of some or all of the occluded pixels in the processed image. For example, if an unwanted object remains stationary for an extended period of time, the ZSL buffer may not contain any images in which the unwanted object is not present. In such cases, the image adjustment engine 545 may utilize inpainting techniques such as those described above with respect to FIG. 3 to replace the occluded pixels. In some cases, the image adjustment engine 545 may combine pixels from multiple images (e.g., image fusion), perform inpainting, any other technique for replacing pixels in an image, and / or any combination thereof.

[0064] In one example embodiment, the image capture system 500 can be configured to use inpainting to remove unwanted objects during the preview and ZSL periods 502. In some embodiments, the image capture system 500 can combine pixels from multiple images for an unwanted object that moved during capture in the ZSL buffer 532 and perform inpainting on any other pixels obscured by the unwanted object. However, any combination of pixel replacement techniques can be used during the preview and ZSL periods 502, the image capture period 504, the image processing period 506, and / or any other operational periods of the image capture device without departing from the scope of this disclosure.

[0065] As described above, the image adjustment system 300 and related techniques described herein can be utilized to remove unwanted objects from images captured by an image capture device (e.g., a camera). Furthermore, the techniques described herein can be used to provide a photographer with a preview image displayed on a display that reflects the scene with the unwanted objects removed. In some cases, unwanted objects can also be automatically identified (e.g., by a machine learning model) and removed from the captured image, thereby obviating the need for post-processing. Furthermore, the systems and techniques described herein can be used to combine pixel information from multiple images stored in an image buffer (e.g., ZSL buffer 532 of FIG. 5 ), allowing the image adjustment system and techniques to generate an accurate reproduction of the captured scene without the unwanted objects. In some cases, other object removal and replacement techniques, such as inpainting, can be used to conserve system resources. For example, inpainting techniques can be used during the preview period (e.g., before capture input is received). In some cases, the preview image may not ultimately be stored on a storage device, and image quality may be sacrificed in favor of low latency so that the preview image is more likely to match the capture image that is processed after the capture input is received.

[0066] 6 is a flow diagram illustrating one embodiment of a process 600 for processing one or more images. At block 602, process 600 includes acquiring a first image of a scene from a camera (e.g., one or more image capture devices 302). In some cases, the scene includes a first object located at a first location and a second object located at a second location (see FIG. 4B).

[0067] At block 604, process 600 includes acquiring a second image of the scene from the camera. In some embodiments, the second image of the scene includes a first object located at a first location and a second object located at a third location, the third location being different from the second location (see FIG. 4E). In some cases, the first image of the scene and the second image of the scene are preview images acquired (e.g., stored in preview buffer 536 of FIG. 5) before acquiring the capture input. In some embodiments, the second image of the scene is an image acquired after acquiring the capture input.

[0068] At block 606, process 600 includes generating (e.g., by image adjustment engine 308) an adjusted second image based on the second image. In some cases, the adjusted second image includes the first object located at the first location, and the second object at the third location is removed from the adjusted second image (see FIG. 4F). In some cases, removing the second object from the adjusted second image includes interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene. In some examples, removing the second object from the adjusted second image includes taking pixels of the first image of the scene associated with the third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0069] At block 608, process 600 includes displaying the adjusted second image on a display (e.g., preview display 550 of FIG. 5). In some embodiments, the first image of the scene and the adjusted second image comprise images (e.g., preview images) captured independently of obtaining a capture input.

[0070] In some embodiments, process 600 includes acquiring a third image of the scene from a camera, acquiring a capture input, and generating a capture image based on the third image of the scene based on the acquiring the capture input.

[0071] In some implementations, process 600 includes, after displaying a first image of the scene on a display, obtaining input associated with selecting a second object for removal from the scene. In some examples, the input associated with selecting the second object for removal from the scene includes at least one or more of physical contact, a gesture, a gaze direction, input from an input device, or voice input. In some examples, process 600 includes obtaining input associated with selecting a first object as a target object for inclusion in the adjusted second image. In some implementations, the input associated with selecting the first object as a target object for inclusion in the adjusted second image includes at least one or more of physical contact, a gesture, a gaze direction, input from an input device, or voice input.

[0072] In some cases, process 600 includes determining (e.g., by object identification engine 304 of FIG. 3 ) that the second object is an unwanted object. In some cases, determining that the second object is an unwanted object includes at least one or more of determining that the second object moved between the first image of the scene and the second image of the scene, determining that the first object is a previously photographed object and that the second object is not a previously photographed object, determining that the depth of the first object and the depth of the second object differ by more than a threshold depth difference amount, determining that the second object is outside of a region of interest based on gaze direction, or determining that audio input associated with the second object indicates an unwanted object.

[0073] In some examples, process 600 includes obtaining a capture input. In some examples, process 600 includes obtaining a third image of the scene. In some aspects, the third image of the scene includes the first object at the first location and the second object. In some cases, process 600 includes determining a fourth location of the second object in the scene. In some implementations, process 600 includes generating an adjusted third image including the first object at the first location. In some cases, the second object is removed from the adjusted third image. In some examples, process 600 includes storing the adjusted third image. In some implementations, process 600 includes obtaining a segmentation (e.g., by feature segmentation engine 306 of FIG. 3 ) associated with the second object and the first image of the scene, associating the third image of the scene with the first image of the scene, and applying the segmentation associated with the second object and the first image of the scene to the third image of the scene. In some embodiments, the first image has a first resolution (e.g., a preview image in preview buffer 536 of FIG. 5 ), and the third image has a second resolution (e.g., an image in ZSL buffer 532 of FIG. 5 ), where the second resolution is different from the first resolution. In some embodiments, generating the adjusted second image includes performing an inpainting process to remove the second object from the second image of the scene, and generating the adjusted third image includes combining pixels of the fourth image of the scene associated with the fourth location with pixels of the third image of the scene. In some embodiments, the pixels of the fourth image of the scene associated with the fourth location include a portion of the scene obscured by the second object in the third image of the scene.

[0074] In some cases, process 600 includes acquiring a fourth image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location. In some examples, the fourth image of the scene includes an image captured independent of acquiring the capture input. In some implementations, process 600 includes generating an adjusted fourth image based on the fourth image. In some examples, the adjusted fourth image includes the first object located at the first location, and the second object is removed from the adjusted fourth image. In some aspects, process 600 includes displaying the adjusted fourth image on a display. In some examples, process 600 includes acquiring input associated with removing the second object based on the first image of the scene. In some implementations, process 600 includes acquiring a fifth image of the scene from a camera, acquiring the capture input, and generating a capture image based on the fifth image of the scene based on acquiring the capture input. In some examples, the captured image includes a first object located at a first location, and a second object located at a second location is removed from the captured image. In some implementations, the process 600 includes storing the captured image.

[0075] In some examples, the processes described herein (e.g., process 600 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more of the processes may be performed by image adjustment system 300 of FIG. 3. In another example, one or more of the processes may be performed by computing system 900 shown in FIG. 9. For example, a computing device having computing system 900 shown in FIG. 9 may include components of image adjustment system 300 and may perform operations of process 600 of FIG. 6 and / or other processes described herein.

[0076] The computing device may include any suitable device, such as a vehicle or a vehicle's computing device (e.g., a vehicle's driver monitoring system (DMS)), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having the resource capabilities to perform the processes described herein, including process 600 and / or other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0077] Components of a computing device may be implemented in circuitry. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0078] Process 600 is illustrated as a logical flow diagram, whose operations represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0079] Furthermore, process 600 and / or other processes described herein can be executed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or a combination thereof. As mentioned above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0080] As described above, various aspects of the present disclosure may employ machine learning models or systems. FIG. 7 illustrates an exemplary implementation of a deep learning neural network 700 that may be used to implement the machine learning-based feature segmentation, instance segmentation, depth estimation, and / or classification described above. The input layer 720 includes input data. In one exemplary implementation, the input layer 720 may include data representing pixels of an input image. The neural network 700 includes multiple hidden layers 722a, 722b, 722n, and 722b. The hidden layers 722a, 722b, 722n may include "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers may be as many as required for a given application. The neural network 700 further includes an output layer 721 that provides output resulting from the processing performed by the hidden layers 722a, 722b, 722n. In one example implementation, the output layer 721 can provide a classification for objects in an input image (e.g., the first input image 410 of FIG. 4A, the second input image 440 of FIG. 4D), which can include classes that identify types of activities (e.g., looking up, looking down, closing eyes, yawning, etc.).

[0081] Neural network 700 is a multi-layered neural network of interconnected nodes. Each node can represent one piece of information. The information associated with a node is shared among different layers, and each layer retains the information as it is processed. In some cases, neural network 700 may include a feed-forward network, in which there are no feedback connections, where the output of the network is fed back into itself. In some cases, neural network 700 may include a recurrent neural network, which may have loops that allow information to be conveyed between nodes while reading at the input.

[0082] Information can be exchanged between nodes via node-to-node interconnections between the various layers. Nodes in the input layer 720 can activate a set of nodes in the first hidden layer 722a. For example, as shown, each input node in the input layer 720 is connected to a respective node in the first hidden layer 722a. The nodes in the first hidden layer 722a can transform each input node's information by applying an activation function to the input node information. The information derived from that transformation can then be passed to nodes in the next hidden layer 722b, which can activate those nodes, which can perform their specified functions. Exemplary functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of the hidden layer 722b can then activate nodes in the next hidden layer, and so on. The output of the last hidden layer 722n can activate one or more nodes in the output layer 721, which provide the output. In some cases, a node in neural network 700 (e.g., node 726) is shown as having multiple output lines, but the node has a single output, and all lines shown as outputting from the node represent the same output value.

[0083] In some cases, each node, or the interconnections between nodes, may have weights, which are sets of parameters derived from training the neural network 700. Once the neural network 700 is trained, it may be referred to as a trained neural network and may be used to classify one or more activities. For example, the interconnections between nodes may represent a piece of information learned about the nodes that are interconnected. The interconnections may have adjustable numerical weights that can be adjusted (e.g., based on a training data set), allowing the neural network 700 to be adaptive to inputs and to learn more as more data is processed.

[0084] Neural network 700 is pre-trained to process features from data in input layer 720 using various hidden layers 722a, 722b, through 722n to provide output via output layer 721. In an embodiment in which neural network 700 is used to identify features in images, neural network 700 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating a feature in the image (for a feature segmentation machine learning system) or a label indicating a class of activity in each image. In an embodiment using object classification for illustrative purposes, the training images can include an image of the number 2, where the label for that image can be [0 0 1 0 0 0 0 0 0 0].

[0085] In some cases, neural network 700 may adjust node weights using a training process called backpropagation. As described above, backpropagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process may be repeated for a certain number of iterations for each set of training images until neural network 700 is well trained so that the layer weights are accurately adjusted.

[0086] For an embodiment for identifying objects in an image, a forward pass may include passing a training image through neural network 700. Before neural network 700 is trained, the weights are first randomized. As an illustrative example, an image may include an array of numbers representing pixels in the image. Each number in the array may include a value between 0 and 255 that describes the pixel intensity at that location in the array. In one embodiment, the array may include a 28x28x3 array of numbers, with 28 rows and 28 columns of pixels and three color components (such as a red component, a green component, and a blue component, or a luma component and two saturation components).

[0087] As mentioned above, for the first training iteration for the neural network 700, due to the randomly selected weights during initialization, the output will likely include values that do not favor any particular class. For example, if the output is a vector with probabilities that an object contains various classes, the probability values for each of the various classes may be equal or at least very similar (e.g., for 10 possible classes, each class may have a probability value of 0.1). With these initial weights, the neural network 700 is unable to determine low-level features and therefore is unable to accurately determine what the object's classification may be. A loss function can be used to analyze the error in the output. Any suitable loss function definition can be used, such as cross-entropy loss. Another example of a loss function is:

[0088]

number

[0089] For the first training images, the actual values will be significantly different from the predicted output, resulting in a large loss (or error). The goal of training is to minimize the amount of loss so that the predicted output is the same as the training labels. The neural network 700 can perform a backward pass by determining which inputs (weights) contributed most to the network's loss, and adjust the weights so that the loss is reduced and ultimately minimized. To determine which weights contributed most to the network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight in a particular layer) can be calculated. After calculating the derivative, a weight update can be performed by updating all of the filter's weights. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be performed by:

[0090]

number

[0091] Neural network 700 may include any suitable deep network. One example is a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input and output layers. The hidden layers of a CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. Neural network 700 may also include any other deep network other than a CNN, such as autoencoders, deep belief nets (DBNs), recurrent neural networks (RNNs), among others.

[0092] FIG. 8 is an exemplary embodiment of a convolutional neural network (CNN) 800. The input layer 820 of the CNN 800 contains data representing an image or frame. For example, this data may include an array of numbers representing pixels of the image, with each number in the array containing a value between 0 and 255 that describes the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two saturation components). The image may be passed through a convolutional hidden layer 822 a, an optional nonlinear activation layer, a pooling hidden layer 822 b, and a fully connected hidden layer 822 c to obtain an output at the output layer 824. 8, only one of each hidden layer is shown, those skilled in the art will understand that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 800. As previously mentioned, the output may indicate a single class of object, or may include a probability of the class that best describes the object in the image.

[0093] The first layer of the CNN 800 is a convolutional hidden layer 822a. The convolutional hidden layer 822a analyzes the image data in the input layer 820. Each node in the convolutional hidden layer 822a is connected to a region of nodes (pixels) in the input image, called the receptive field. The convolutional hidden layer 822a can be thought of as one or more filters (each filter corresponding to a different activation map or feature map), with each convolutional iteration of a filter being a node or neuron in the convolutional hidden layer 822a. For example, the region of the input image covered by a filter in each convolutional iteration is the receptive field for that filter. In one exemplary implementation, if the input image includes a 28x28 array and each filter (and corresponding receptive field) is a 5x5 array, there will be 24x24 nodes in the convolutional hidden layer 822a. Each node learns to analyze its particular local receptive field in the input image by learning a weight and, in some cases, a global bias for each connection between the node and the receptive field associated with that node. Each node in the hidden layer 822a will have the same weights and biases (called shared weights and shared biases). For example, the filter has an array of weights (numbers) and a depth equal to the input. For the example of an image frame, the filter will have a depth of 3 (according to the three color components of the input image). The size of an example example filter array is 5x5x3, corresponding to the size of the receptive field of the node.

[0094] The convolutional nature of the convolutional hidden layer 822a results from each node of the convolutional layer being applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 822a can start at the upper left corner of the input image array and convolve around the input image. As described above, each convolutional iteration of the filter can be considered a node or neuron of the convolutional hidden layer 822a. In each convolutional iteration, the filter value is multiplied by the corresponding numerical value of the original pixel values of the image (e.g., a 5x5 filter array is multiplied by a 5x5 array of input pixel values in the upper left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain a total for that iteration or node. This process then continues at the next location in the input image according to the receptive field of the next node in the convolutional hidden layer 822a. For example, the filter can be moved to the next receptive field in a certain step amount (called a stride). The stride can be set to 1 or any other suitable amount. For example, if the stride is set to 1, the filter will be moved one pixel to the right in each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result for that location, which in turn determines a sum value for each node in the convolutional hidden layer 822a.

[0095] The mapping from the input layer to the convolutional hidden layer 822a is called an activation map (or feature map). The activation map contains a value for each node that represents the filter result at each location in the input volume. The activation map may include an array containing the various sum values resulting from each iteration of the filter on the input volume. For example, if a 5x5 filter is applied to each pixel of a 28x28 input image (with a stride of 1), the activation map would include a 24x24 array. The convolutional hidden layer 822a may include several activation maps to identify multiple features within an image. The example shown in Figure 8 includes three activation maps. Using the three activation maps, the convolutional hidden layer 822a can detect three different types of features, each detectable across the entire image.

[0096] In some embodiments, a nonlinear hidden layer can be applied after the convolutional hidden layer 822a. A nonlinear layer can be used to introduce nonlinearity into a system that would otherwise compute a linear operation. An exemplary embodiment of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply a function f(x)=max(0,x) to all of the values in the input volume, which changes all negative activations to 0. Therefore, the ReLU can increase the nonlinear characteristics of the CNN 800 without affecting the receptive field of the convolutional hidden layer 822a.

[0097] A pooling hidden layer 822b may be applied after the convolutional hidden layer 822a (and, if used, after the nonlinear hidden layer). The pooling hidden layer 822b is used to simplify the information in the output from the convolutional hidden layer 822a. For example, the pooling hidden layer 822b may take each activation map output from the convolutional hidden layer 822a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2-norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 822a. A pooling function (e.g., a max pooling filter, an L2-norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 822a. In the example shown in FIG. 8, three pooling filters are used for the three activation maps in the convolutional hidden layer 822a.

[0098] In some embodiments, max pooling can be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the filter dimensions, such as a stride of 2) to the activation map output from the convolutional hidden layer 822a. The output from the max pooling filter contains the maximum numerical value in any subregion that the filter convolves around. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes (each node is a value in the activation map) in the previous layer. For example, four values (nodes) in the activation map would be analyzed by a 2×2 max pooling filter at each iteration of the filter, and the maximum value of those four values would be output as the “max” value. If such a max pooling filter is applied to an activation filter from the convolutional hidden layer 822a, which has dimensions of 24×24 nodes, the output from the pooling hidden layer 822b would be an array of 12×12 nodes.

[0099] In some embodiments, an L2 norm pooling filter can also be used, which involves calculating the square root of the sum of the squares of the values in a 2x2 region (or other suitable region) of the activation map (rather than calculating the maximum value as is done in max pooling) and using that calculated value as the output.

[0100] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling function) determines whether a given feature is found anywhere within a region of the image, and the exact location information is discarded. This can be done without affecting the outcome of feature detection, because once a feature is found, the exact location of the feature is less important than its approximate location relative to other features. Max pooling (as well as other pooling methods) offers the advantage of pooling far fewer features, thus reducing the number of parameters required in subsequent layers of the CNN 800.

[0101] The final layer of connections in the network is a fully connected layer that connects every node from the pooling hidden layer 822b to every single output node in the output layer 824. Using the above example, the input layer includes 28x28 nodes encoding pixel intensities of the input image, the convolutional hidden layer 822a includes 3x24x24 hidden feature nodes based on applying 5x5 local receptive fields (for the filters) to three activation maps, and the pooling hidden layer 822b includes a layer of 3x12x12 hidden feature nodes based on applying max-pooling filters to 2x2 regions across each of the three feature maps. Extending this example, the output layer 824 could include 10 output nodes. In such an example, every node in the 3x12x12 pooling hidden layer 822b is connected to every node in the output layer 824.

[0102] The fully connected layer 822c can take the output of the previous pooling hidden layer 822b (which should represent the activation map of high-level features) and determine the features that are most correlated to a particular class. For example, the fully connected layer 822c can determine the high-level features that are most strongly correlated to a particular class and can include weights (nodes) for the high-level features. By calculating the product between the weights of the fully connected layer 822c and the weights of the pooling hidden layer 822b, probabilities for various classes can be obtained. For example, if the CNN 800 is being used to predict that an object in an image is a person, there will be high values in the activation map that represent high-level features of a person (e.g., the presence of two legs, the presence of a face on top of the object, the presence of two eyes on the top left and top right of the face, the presence of a nose in the center of the face, the presence of a mouth at the bottom of the face, and / or other features common to people).

[0103] In some embodiments, the output from the output layer 824 may include an M-dimensional vector (in the preceding embodiment, M=10). M indicates the number of classes the CNN 800 must choose from when classifying objects in an image. Other exemplary outputs may also be provided. Each number in the M-dimensional vector may represent the probability that the object is of a particular class. In one exemplary embodiment, if a 10-dimensional output vector representing 10 different object classes is [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is a third class object (e.g., a dog), an 80% probability that the image is a fourth class object (e.g., a human), and a 15% probability that the image is a sixth class object (e.g., a kangaroo). The probabilities associated with a class may be considered a level of confidence that the object is part of that class.

[0104] Figure 9 illustrates an example of a system for implementing certain aspects of the present technology. In particular, Figure 9 illustrates an example of a computing system 900, which may be any computing device comprising, for example, an internal computing system, a remote computing system, a camera, or any component thereof, where the components of the system communicate with each other using a connection 905. The connection 905 may be a physical connection using a bus or a direct connection to a processor 910, such as in a chipset architecture. The connection 905 may also be a virtual connection, a network connection, or a logical connection.

[0105] In some embodiments, computing system 900 is a distributed system, allowing the functionality described in this disclosure to be distributed across one data center, multiple data centers, within a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functionality described with respect to that component. In some embodiments, these components may be physical or virtual devices.

[0106] The exemplary computing system 900 includes at least one processing unit (CPU or processor) 910 and connections 905 coupling various system components to the processor 910, including system memory 915, such as read-only memory (ROM) 920 and random access memory (RAM) 925. The computing system 900 may include a cache 912 of high-speed memory connected directly to the processor 910, connected in close proximity to the processor 910, or integrated as part of the processor 910.

[0107] Processor 910 may include any general-purpose processor, hardware or software services, such as services 932, 934, and 936 stored in storage device 930, configured to control processor 910, and special-purpose processors where software instructions are embedded in the actual processor design. Processor 910 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0108] To enable user interaction, computing system 900 includes input devices 945, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 900 may also include output devices 935, which may be one or more of a number of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with computing system 900. Computing system 900 may include a communication interface 940, which may generally govern and manage user input and system output.The communication interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, an optical fiber port / plug, a proprietary wired port / plug, a BLUETOOTH® wireless signal transmission, a BLUETOOTH® low energy (BLE) wireless signal transmission, an IBEACON® wireless signal transmission, a radio-frequency identification (RFID) wireless signal transmission, a near-field communications (NFC) wireless signal transmission, a dedicated short range communication (DSRC) wireless signal transmission, an 802.11 Wi-Fi wireless signal transmission, a wireless local area network (WLAN) signal transmission, a visible light communication (VLC), a Worldwide Interoperability for Microwave Access (WiMAX), an infrared (IR) communication wireless signal transmission, a public switched telephone network (PSTN) wireless signal transmission, a The device may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing PSTN (Switched Telephone Network) signal transmission, ISDN (Integrated Services Digital Network) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.Communications interface 940 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of computing system 900 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems, including, but not limited to, the U.S.-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no constraint to operating on any particular hardware configuration, and therefore, the basic features herein can be easily substituted for improved hardware or firmware configurations as they are developed.

[0109] The storage device 930 can be a non-volatile and / or non-transitory and / or computer-readable memory device, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM / ReRAM), phase change memory (PRAM),The memory may be a hard disk or other type of computer-readable medium capable of storing data that is accessible by a computer, such as PCM, spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof;

[0110] The storage device 930 may include software services, servers, services, etc., where code defining such software, when executed by the processor 910, causes the processor to perform functions in the system. In some embodiments, hardware services that perform particular functions may include software components stored in a computer-readable medium in association with the necessary hardware components, such as the processor 910, connections 905, output devices 935, etc., to perform the functions.

[0111] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may include non-transitory media capable of storing data, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions, which may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0112] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals, including bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0113] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including functional blocks that include devices, device components, steps or routines in methods embodied in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0114] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0115] The processes and methods according to the above-described embodiments can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions, or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during methods according to the described embodiments include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0116] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical example form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in a peripheral device or add-in card. Such functionality may also be implemented among various chips on a circuit board or among various processes running within a single device, as further examples.

[0117] The instructions, media for carrying such instructions, computing resources for executing those instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0118] While the foregoing description describes aspects of the present application with reference to specific embodiments thereof, those skilled in the art will recognize that the present application is not limited thereto. Therefore, while exemplary embodiments of the present application have been described in detail herein, it should be understood that, except as limited by the prior art, the concepts of the present application may be variously embodied and employed in other ways, and the appended claims are intended to be construed to include such variations. The various features and aspects of the present application described above may be used individually or in combination. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the scope of the present application. Accordingly, the specification and drawings should be regarded as illustrative and not restrictive. For illustrative purposes, methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0119] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0120] Where a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0121] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection, and / or other suitable communication interface).

[0122] Claim language or other language referring to "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, a claim language referring to "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, a claim language referring to "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language referring to "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may also include items not recited within the set of A and B.

[0123] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0124] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. These technologies may also, or alternatively, be implemented at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0125] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0126] Exemplary aspects of the present disclosure include: Aspect 1: An apparatus for processing one or more images. The apparatus includes a memory (e.g., implemented in a circuit) and a processor (or multiple processors) coupled to the memory. The processor is configured to: acquire a first image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location; acquire a second image of the scene from the camera, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; generate an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and display the adjusted second image on a display, the first image of the scene and the adjusted second image including images captured independent of obtaining a capture input.

[0127] Aspect 2: The device of aspect 1, wherein the processor is configured to acquire a third image of the scene from the camera, acquire a capture input, and generate a capture image based on the third image of the scene based on acquiring the capture input, the capture image including a first object located at a first position, and a second object located at a third position being removed from the capture image.

[0128] Aspect 3: The apparatus of any of aspects 1 and 2, wherein the second image of the scene and the third image of the scene are associated with a common identifier.

[0129] Aspect 4: The device of any of aspects 1 to 3, wherein the third image of the scene includes an image acquired before the capture input, and the third image of the scene is stored in at least one memory.

[0130] Aspect 5: The apparatus of any of aspects 1-4, wherein the at least one memory includes a zero shutter lag (ZSL) buffer.

[0131] Aspect 6: The device of any of aspects 1-5, wherein the processor is configured to obtain an input associated with selecting a second object for removal from the scene.

[0132] Aspect 7: The device of any of aspects 1 to 6, wherein the processor is configured to acquire input associated with selecting the first object as a target object for inclusion in the adjusted second image.

[0133] Aspect 8: The device of any of aspects 1 to 7, wherein the input associated with selecting the first object as a target object for inclusion in the adjusted second image includes at least one or more of physical contact, gesture, gaze direction, input from an input device, or voice input.

[0134] Aspect 9: The device of any of aspects 1 to 8, wherein the processor is configured to determine that the second object is an unwanted object, and determining that the second object is an unwanted object includes at least one or more of determining that the second object has moved between the first image of the scene and the second image of the scene, determining that the first object is a previously photographed object and that the second object is not a previously photographed object, or determining that the depth of the first object and the depth of the second object differ by more than a threshold depth difference amount.

[0135] Aspect 10: The apparatus of any of aspects 1 to 9, wherein removing the second object from the adjusted second image includes interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

[0136] Embodiment 11: The apparatus of any of embodiments 1 to 10, wherein removing the second object from the adjusted second image includes obtaining pixels of the first image of the scene associated with a third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0137] Aspect 12: The device of any of aspects 1 to 11, wherein the first image of the scene and the second image of the scene are preview images acquired before acquiring the capture input.

[0138] Aspect 13: The device of any of aspects 1 to 12, wherein the second image of the scene is an image acquired after obtaining the capture input.

[0139] Aspect 14: The apparatus of any of aspects 1-13, wherein the processor is configured to: acquire a capture input; acquire a third image of the scene, the third image of the scene including a first object at a first position and a second object; determine a fourth position of the second object within the scene; generate an adjusted third image including the first object at the first position, the second object being removed from the adjusted third image; and store the third image.

[0140] Aspect 15: The apparatus of any of aspects 1 to 14, wherein the processor is configured to obtain a segmentation associated with the second object and the first image of the scene, associate a third image of the scene with the first image of the scene, and apply the segmentation associated with the second object and the first image of the scene to the third image of the scene.

[0141] Aspect 16: The device of any of aspects 1 to 15, wherein the first image has a first resolution, the third image has a second resolution, and the second resolution is different from the first resolution.

[0142] Embodiment 17: The apparatus of any of embodiments 1 to 16, wherein to generate the adjusted second image, at least one processor is configured to perform an inpainting process to remove the second object from the second image of the scene, and to generate the adjusted third image, the adjusted third image includes combining pixels of the fourth image of the scene associated with a fourth location with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth location include a portion of the scene in the third image of the scene that is obscured by the second object.

[0143] Embodiment 18: The device of any of embodiments 1 to 17, wherein the third image and the fourth image are stored in a ZSL buffer.

[0144] Aspect 19: The device of any of aspects 1-18, wherein the processor is configured to: acquire a fourth image of a scene from the camera, the scene including a first object located at a first position and a second object located at a second position, the fourth image of the scene including an image captured independent of obtaining a capture input; generate an adjusted fourth image based on the fourth image, the adjusted fourth image including the first object located at the first position and the second object removed from the adjusted fourth image; and display the adjusted fourth image on the display.

[0145] Example 20: The apparatus of any of Examples 1-19, wherein the processor is configured to obtain an input associated with removing a second object based on a first image of the scene.

[0146] Embodiment 21: The apparatus of any of embodiments 1 to 20, wherein the second object is removed based on an inpainting process.

[0147] Aspect 22: The device of any of aspects 1 to 21, wherein the processor is configured to acquire a fifth image of the scene from the camera, acquire a capture input, generate a capture image based on the fifth image of the scene based on the acquisition of the capture input, the capture image including a first object located at a first position and a second object located at a second position being removed from the capture image, and store the capture image in at least one memory.

[0148] Aspect 23. A method for processing one or more images, the method comprising: acquiring a first image of a scene from a camera, the scene including a first object located at a first location and a second object located at a second location; acquiring a second image of the scene from the camera, the second image of the scene including the first object located at the first location and the second object located at a third location, the third location being different from the second location; generating an adjusted second image based on the second image, the adjusted second image including the first object located at the first location and the second object at the third location removed from the adjusted second image; and displaying the adjusted second image on a display, the first image of the scene and the adjusted second image including images captured independent of obtaining a capture input.

[0149] Aspect 24. The method of aspect 23, further comprising: acquiring a third image of the scene from the camera; acquiring a capture input; and generating a capture image based on the third image of the scene based on acquiring the capture input, the capture image including a first object located at a first position, and a second object located at a third position being removed from the capture image.

[0150] Embodiment 25. The method of any of embodiments 23, 24, wherein the second image of the scene and the third image of the scene are associated with a common identifier.

[0151] Embodiment 26. The method of any of embodiments 23-25, wherein the third image of the scene includes an image acquired before the capture input, and the third image of the scene is stored in the memory.

[0152] Embodiment 27. The method of any of embodiments 23-26, wherein the memory includes a zero shutter lag (ZSL) buffer.

[0153] Embodiment 28. The method of any of embodiments 23 to 27, further comprising, after displaying a first image of the scene on the display, acquiring input associated with selecting a second object for removal from the scene.

[0154] Embodiment 29. The method of any of embodiments 23-28, further comprising obtaining input associated with selecting the first object as a target object for inclusion in the adjusted second image.

[0155] Embodiment 30. The method of any of embodiments 23-29, wherein the input associated with selecting the first object as the target object for inclusion in the adjusted second image includes at least one or more of physical contact, gesture, gaze direction, input from an input device, or voice input.

[0156] Aspect 31. The method of any of aspects 23-30, further comprising determining that the second object is an unwanted object, wherein determining that the second object is an unwanted object comprises at least one or more of: determining that the second object has moved between the first image of the scene and the second image of the scene; determining that the first object is a previously photographed object and that the second object is not a previously photographed object; or determining that the depth of the first object and the depth of the second object differ by more than a threshold depth difference amount.

[0157] Embodiment 32. The method of any of embodiments 23 to 31, wherein removing the second object from the adjusted second image includes interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

[0158] Embodiment 33. The method of any of embodiments 23-32, wherein removing the second object from the adjusted second image includes obtaining pixels of the first image of the scene associated with a third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0159] Embodiment 34. The method of any of embodiments 23-33, wherein the first image of the scene and the second image of the scene are preview images acquired prior to acquiring the capture input.

[0160] Embodiment 35. The method of any of embodiments 23-34, wherein the second image of the scene is an image acquired after obtaining the capture input.

[0161] Embodiment 36. The method of any of embodiments 23-35, further comprising: acquiring a capture input; acquiring a third image of the scene, the third image of the scene including a first object at a first position and a second object; determining a fourth position of the second object within the scene; generating an adjusted third image including the first object at the first position, the second object being removed from the adjusted third image; and storing the third image.

[0162] Embodiment 37. The method of any of embodiments 23-36, further including obtaining a segmentation associated with a second object and the first image of the scene, associating a third image of the scene with the first image of the scene, and applying the segmentation associated with the second object and the first image of the scene to the third image of the scene.

[0163] Embodiment 38. The method of any of embodiments 23-37, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution.

[0164] Embodiment 39. The method of any of embodiments 23-38, wherein generating the adjusted second image includes performing an inpainting process to remove the second object from the second image of the scene, and generating the adjusted third image includes combining pixels of the fourth image of the scene associated with a fourth location with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth location include a portion of the scene in the third image of the scene that is obscured by the second object.

[0165] Embodiment 40. The method of any of embodiments 23 to 39, wherein the third image and the fourth image are stored in a ZSL buffer.

[0166] Embodiment 41. The method of any of embodiments 23-40, further including: acquiring a fourth image of a scene from a camera, the scene including a first object located at a first position and a second object located at a second position, the fourth image of the scene including an image captured independent of obtaining a capture input; generating an adjusted fourth image based on the fourth image, the adjusted fourth image including the first object located at the first position and the second object removed from the adjusted fourth image; and displaying the adjusted fourth image on a display.

[0167] Embodiment 42. The method of any of embodiments 23-41, further including obtaining input associated with removing a second object based on a first image of the scene.

[0168] Embodiment 43. The method of any of embodiments 23-42, wherein the second object is removed based on an inpainting process.

[0169] Embodiment 44. The method of any of embodiments 23-43, further including: acquiring a fifth image of the scene from the camera; acquiring a capture input; generating a capture image based on the fifth image of the scene based on acquiring the capture input, the capture image including a first object located at a first position and a second object located at a second position being removed from the capture image; and storing the capture image.

[0170] Aspect 45: A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations of aspects 1 to 44.

[0171] Aspect 46: An apparatus comprising one or more means for performing any of the operations of aspects 1 to 44.

Claims

1. A device for processing one or more images, At least one memory, The at least one processor coupled to the at least one memory and The at least one processor is The process involves obtaining a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position. Acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object located at the first position and the second object located at the third position, and the third position is different from the second position. Based on the fact that the depth of the first object and the depth of the second object differ by a threshold depth difference, it is determined that the second object is an unnecessary object. Based on the determination that the second object is the unnecessary object, a modified second image is generated based on the second image, wherein the modified second image includes the first object located at the first position, and the second object at the third position is removed from the modified second image. Displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring capture input. A device configured to perform the following actions.

2. The aforementioned at least one processor, To acquire a third image of the aforementioned scene from the camera, To acquire the aforementioned capture input, Based on acquiring the capture input, a capture image is generated based on the third image of the scene, wherein the capture image includes the first object located at the first position, and the second object located at the third position is removed from the capture image. It is configured to do the following: The second image of the aforementioned scene and the third image of the aforementioned scene are associated with a common identifier. The third image of the scene includes an image acquired before the capture input, and the third image of the scene is stored in at least one memory. The apparatus according to claim 1, wherein the at least one memory includes a zero shutter lag (ZSL) buffer.

3. The apparatus according to claim 1, wherein the at least one processor is configured to take an input associated with selecting the second object to remove from the scene.

4. The at least one processor is configured to receive an input associated with selecting the first object as a target object to be included in the adjusted second image, The apparatus according to claim 1, wherein the input associated with selecting the first object as a target object to be included in the adjusted second image includes at least one or more of the following: physical contact, gesture, gaze direction, input from an input device, or audio input.

5. In order to determine that the second object is an unnecessary object, the at least one processor, The apparatus according to claim 1, wherein it is configured to determine that the first object is an object that was previously photographed and the second object is not an object that was previously photographed.

6. To remove the second object from the adjusted second image, the at least one processor, The apparatus according to claim 1, configured to interpolate the color of pixels in the second image and generate pixel data relating to one or more pixels associated with the second object in the second image of the scene.

7. To remove the second object from the adjusted second image, the at least one processor, The apparatus according to claim 1, configured to acquire a pixel of the first image of the scene associated with the third position and replace a corresponding pixel of the second image of the scene associated with the third position.

8. The first image and the second image of the scene are preview images acquired before the capture input was acquired, and / or The apparatus according to claim 1, wherein the second image of the scene is an image acquired after acquiring the capture input.

9. The aforementioned at least one processor, To acquire capture input, To obtain a third image of the scene, wherein the third image of the scene includes the first object at the first position and the second object. Determining the fourth position of the second object in the aforementioned scene, The method involves generating a corrected third image that includes the first object at the first position, wherein the second object is removed from the corrected third image. To store the aforementioned third image and The apparatus according to claim 1, configured to perform the following:

10. The aforementioned at least one processor, Obtaining segmentation associated with the second object and the first image of the scene, Associating the third image of the scene with the first image of the scene, Applying the segmentation associated with the second object and the first image of the scene to the third image of the scene It is configured to do the following: The apparatus according to claim 9, wherein the first image has a first resolution, and the third image has a second resolution, the second resolution being different from the first resolution.

11. To generate the adjusted second image, the at least one processor, In order to remove the second object from the second image of the aforementioned scene, an inpainting process is performed, The process involves generating the adjusted third image, wherein the process includes combining pixels of the fourth image of the scene, associated with the fourth position, with pixels of the third image of the scene, such that the pixels of the fourth image of the scene, associated with the fourth position, include a portion of the third image of the scene that is obscured by the second object. The apparatus according to claim 9, configured to perform the following:

12. The aforementioned at least one processor, Acquiring a fourth image of the scene from the camera, wherein the scene includes a first object located at a first position and a second object located at a second position, and the fourth image of the scene includes an image captured independently of acquiring the capture input. The method involves generating an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object located at the first position, and the second object is removed from the adjusted fourth image. Display the adjusted fourth image on the display. The apparatus according to claim 1, configured to perform the following:

13. The apparatus according to claim 12, wherein the at least one processor is configured to acquire an input associated with removing the second object based on the first image of the scene.

14. The aforementioned at least one processor, The fifth image of the aforementioned scene is acquired from the camera, To acquire the aforementioned capture input, Based on acquiring the capture input, a capture image is generated based on the fifth image of the scene, wherein the capture image includes the first object located at the first position, and the second object located at the second position is removed from the capture image. The captured image is stored in at least one memory. The apparatus according to claim 12, configured to perform the following:

15. A method for processing one or more images, The process involves obtaining a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position. Acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object located at the first position and the second object located at the third position, and the third position is different from the second position. Based on the fact that the depth of the first object and the depth of the second object differ by a threshold depth difference, it is determined that the second object is an unnecessary object. Based on the determination that the second object is the unnecessary object, a modified second image is generated based on the second image, wherein the modified second image includes the first object located at the first position, and the second object at the third position is removed from the modified second image. Displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring capture input. Methods that include...

Images