Object removal in image

Abstract

Systems and techniques are provided for adjusting an object in an image. For example, the processing may include acquiring a first image of a scene from a camera. The scene may include a first object at a first location and a second object at a second location. The processing may include acquiring a second image of the scene from the camera. The second image of the scene may include a first object at a first location and a second object at a third location different from the second location. The processing may include generating an adjusted second image based on the second image. The adjusted second image may include a first object at a first position. The second object at the third position is removed from the adjusted second image. The processing may include displaying the adjusted second image on the display.

Description

Technical Field

[0001] The present disclosure generally relates to adjusting or modifying images. For example, aspects of the present disclosure relate to systems and techniques for removing objects from images. Background Art

[0002] Many devices and systems capture a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera can capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed to perform one or more functions, such as output for display, output for processing and / or consumption by other devices, and other uses.

[0003] When using a device to capture an image of a scene, one or more objects may be present in the scene that are not desired in the final captured image. For example, a user may wish to capture an image of a particular visitor in front of a national monument, while many passersby may be present in the scene.

[0004] The user can try to capture an image of the scene at the exact moment when there are no passersby in the scene. However, in some scenarios, a scene without unwanted objects may not appear and / or may require a long waiting period. Summary of the Invention

[0005] Disclosed are systems, devices, methods, and computer-readable media for adjusting objects in an image. According to at least one example, a method for adjusting objects in an image is provided. The method includes: acquiring a first image of a scene from a camera, wherein the scene includes a first object at a first position and a second object at a second position; acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object at the first position and the second object at a third position, the third position being different from the second position; generating an adjusted second image based on the second image, wherein the adjusted second image includes the first object at the first position, and the second object at the third position is removed from the adjusted second image; and displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring a capture input.

[0006] In another example, a device for adjusting an object in an image is provided, the device comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: acquire a first image of a scene from a camera, wherein the scene includes a first object at a first position and a second object at a second position; acquire a second image of the scene from the camera, wherein the second image of the scene includes the first object at the first position and the second object at a third position, the third position being different from the second position; generate an adjusted second image based on the second image, wherein the adjusted second image includes the first object at the first position and the second object at the third position is removed from the adjusted second image; and display the adjusted second image on a display, wherein the first image and the adjusted second image of the scene include images captured independently of acquiring a capture input.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: acquire a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position; acquire a second image of the scene from the camera, wherein the second image of the scene includes the first object located at the first position and the second object located at a third position, the third position being different from the second position; generate an adjusted second image based on the second image, wherein the adjusted second image includes the first object located at the first position and the second object at the third position is removed from the adjusted second image; and display the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring a capture input.

[0008] In another example, an apparatus for adjusting an object in an image is provided. The apparatus includes: means for acquiring a first image of a scene from a camera, wherein the scene includes a first object at a first position and a second object at a second position; means for acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object at the first position and the second object at a third position, the third position being different from the second position; means for generating an adjusted second image based on the second image, wherein the adjusted second image includes the first object at the first position, and the second object at the third position is removed from the adjusted second image; and means for displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring a capture input.

[0009] In some aspects, one or more of the above-mentioned devices is a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other device, is part of, or includes, a device. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the location and / or posture of the device, the state of the device, and / or other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used solely to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and embodiments will become more fully apparent upon reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] An illustrative embodiment of the present application will be described in detail below with reference to the following figures:

[0013] Figures 1A to 1C is a diagram showing example images of different fields of view according to some examples of the present disclosure;

[0014] Figure 2 is a block diagram illustrating the architecture of an image capture and processing device according to some examples of the present disclosure;

[0015] Figure 3 is a block diagram illustrating an example image adjustment system according to some examples of the present disclosure;

[0016] Figures 4A to 4F are images illustrating example image adjustments according to some examples of the present disclosure;

[0017] Figure 5 is a block diagram illustrating an example image capture system including an image adjustment system according to some examples of the present disclosure;

[0018] Figure 6 is a block diagram illustrating an example image adjustment process according to some examples of the present disclosure;

[0019] Figure 7is a block diagram illustrating an example of a deep learning network according to some examples of the present disclosure;

[0020] Figure 8 is a block diagram illustrating an example of a convolutional neural network according to some examples;

[0021] Figure 9 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0022] Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, while some of these aspects and embodiments can be applied in combination, which will be apparent to those skilled in the art. In the subsequent description, for ease of explanation, specific details are listed to provide a thorough understanding of the embodiments of this application. However, it is apparent that various embodiments can be implemented without these specific details. The drawings and description are not restrictive.

[0023] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an implementation description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of application described in the appended claims.

[0024] When an image of a scene is captured, in addition to target objects (such as people, animals, vehicles, structures, etc.), there may also be unwanted objects (such as people, animals, vehicles, etc.) in the scene. Figures 1A to 1C is a diagram illustrating a scene including an unwanted object and a process of removing the unwanted object from an image of the scene. Figure 1A is an image 100 of two people 102 sitting on stairs. In the image 100, the people 102 are the target objects (e.g., subjects) that are desired to be captured in the scene of the image 100. Figure 1A , image 100 also includes three persons 104A, 104B, 104C located behind person 102. In some cases, it may be desirable to remove persons 104A, 104B, 104C in the background of image 100.

[0025] Figure 1B shows a segmented image 110, Figure 1A106C are depicted by segmentation contours 106A, 106B, 106C. For example, the three persons 104A, 104B, 104C may be selected (e.g., by user input, a machine learning model, etc.) and identified for removal from the segmented image 110. In some cases, a segmentation process may be used to determine which pixels in the segmented image 110 belong to the respective persons 104A, 104B, 104C indicated by the segmentation contours 106A, 106B, 106C. In some cases, all or part of the segmentation shown in the segmented image 110 may be performed by a machine learning model. For example, a machine learning model may be trained to detect features in the segmented image 110. In some cases, based on the detected features in the segmented image 110, the machine learning model may classify objects in the segmented image 110. Example classifications for the persons 104A, 104B, 104C may include "person," "background object," and the like. In some cases, a user may select which objects (or categories) in segmented image 110 to target for adjustment (eg, magnification).

[0026] In some aspects, adjusting the image 100 by removing the persons 104A, 104B, 104C may leave blank areas in the resulting adjusted image. Figure 1C An adjusted image 120 is shown, in which the blank areas left by the removal of persons 104A, 104B, and 104C are filled with area 108. In the example shown, filled area 108 is shown as including sky and clouds consistent with the background. In some cases, portions of adjusted image 120 may be filled using an inpainting process. For example, the inpainting process may attempt to fill the blank areas of adjusted image 120 with a color that approximates (or estimates) the color of the portion of the captured scene in image 100 that was occluded by persons 104A, 104B, and 104C. In one illustrative example, inpainting may be performed by interpolating and / or blending colors from pixels adjacent to the blank areas in input image 100 and applying the interpolated colors to the blank areas. In some cases, after replacing the blank areas with the estimated color values, the outlines of the removed objects (e.g., persons 104A, 104B, and 104C) may be visible. In some cases, image adjustment techniques (eg, inpainting processing) may perform blending and smoothing around the edges of the persons 104A, 104B, 104C (eg, along the segmentation outlines 106A, 106B, 106C).

[0027] In another illustrative example, one or more additional images (not shown) of the same scene or portion of the same scene as segmented image 110 may include portions of the scene that were occluded by people 104A, 104B, 104C in segmented image 110. In some examples, the blank areas created by removing unwanted objects 104A, 104B, 104C in segmented image 110 can be filled by combining pixels from the one or more additional images with image 100, adjusted image 120, one or more additional images, or any combination thereof. However, in some cases, the one or more additional images of the scene or portion of the scene may not have been captured and / or stored. Therefore, techniques for combining pixels from multiple images may not be available.

[0028] During the image capture process, systems and techniques are needed to identify objects to be removed from an image and remove the identified objects. For example, before capturing an image of a scene (e.g., before receiving a capture input), unwanted objects appearing in a preview image (e.g., displayed on a display) can be identified for removal. Thus, the preview image can be matched with a final captured image, which is stored in a memory and similarly has the unwanted objects removed.

[0029] The systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") described herein are used to identify objects to be removed from an image during image capture and to remove the identified objects. In some cases, after one or more unwanted objects are identified to be removed, an additional preview image can be adjusted to remove the unwanted objects before capture. In some examples, when an image of a scene is captured (e.g., in response to a capture input), the unwanted objects can be removed from the scene before the adjusted image is stored in storage. For example, a photographer can see a preview image showing the unwanted objects removed before pressing the shutter (or providing any other type of capture input).

[0030] Various aspects of the technology described herein will be discussed below in conjunction with the accompanying drawings. Figure 2 is a block diagram illustrating the architecture of image capture and processing system 200. Image capture and processing system 200 includes various components for capturing and processing images of a scene, such as an image of scene 210. Image capture and processing system 200 can capture individual images (or photographs) and / or can capture a video comprising a particular sequence of multiple images (or video frames). Lens 215 of image capture and processing system 200 faces scene 210 and receives light from scene 210. Lens 215 refracts light toward image sensor 230. Light received by lens 215 passes through an aperture controlled by one or more control mechanisms 220 and is received by image sensor 230.

[0031] The one or more control mechanisms 220 can control exposure, focus, and / or zoom based on information from the image sensor 230 and / or based on information from the image processor 250. The one or more control mechanisms 220 can include multiple mechanisms and components; for example, the one or more control mechanisms 220 can include one or more exposure control mechanisms 225A, one or more focus control mechanisms 225B, and / or one or more zoom control mechanisms 225C. In addition to the control mechanisms shown in the figure, the one or more control mechanisms 220 can also include additional control mechanisms, such as control mechanisms for controlling analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0032] Focus control mechanism 225B of control mechanism 220 can obtain a focus setting. In some examples, focus control mechanism 225B stores the focus setting in a memory register. Based on the focus setting, focus control mechanism 225B can adjust the position of lens 215 relative to image sensor 230. For example, based on the focus setting, focus control mechanism 225B can move lens 215 closer to or farther from image sensor 230 by driving a motor or servo (or other lens mechanism) to adjust the focus. In some cases, image capture and processing system 200 may include additional lenses, such as one or more microlenses on each photodiode of image sensor 230, each microlens refracting light received from lens 215 toward a corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting can be determined using control mechanism 220, image sensor 230, and / or image processor 250. Focus settings may be referred to as image capture settings and / or image processing settings.

[0033] Exposure control mechanism 225A of control mechanism 220 can obtain an exposure setting. In some cases, exposure control mechanism 225A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 225A can control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 230 (e.g., ISO speed or film speed), the analog gain applied by image sensor 230, or any combination thereof. The exposure setting can be referred to as an image capture setting and / or an image processing setting.

[0034] Zoom control mechanism 225C of control mechanism 220 acquires a zoom setting. In some examples, zoom control mechanism 225C stores the zoom setting in a memory register. Based on the zoom setting, zoom control mechanism 225C can control the focus distance of a lens assembly (lens assembly) comprising lens 215 and one or more additional lenses. For example, zoom control mechanism 225C can control the focus distance of the lens assembly by driving one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly can include a focusing lens (which in some cases can be lens 215) that first receives light from scene 210, and then passes through an afocal zoom system between the focusing lens (e.g., lens 215) and image sensor 230 before reaching image sensor 230. In some cases, a non-focus zoom system can include two positive lenses (e.g., focusing, convex lenses) with equal or similar focus distances (e.g., within a threshold difference of each other), with a negative lens (e.g., defocusing, concave lens) between them. In some cases, the zoom control mechanism 225C moves one or more lenses in the non-focus zoom system, such as a negative lens and one or two positive lenses.

[0035] Image sensor 230 includes an array of one or more photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 230. In some cases, different photodiodes may be covered by different color filters and, therefore, measure light that matches the color filters covering the photodiodes. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, with each pixel of an image generated based on red light data from at least one photodiode covered by the red filter, blue light data from at least one photodiode covered by the blue filter, and green light data from at least one photodiode covered by the green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to the red, blue, and / or green filters. Some image sensors (such as image sensor 230) may not use color filters at all, but rather may use different photodiodes (in some cases stacked vertically) throughout the pixel array. The different photodiodes throughout the pixel array may have different spectral sensitivity curves and, therefore, respond to different wavelengths of light. A monochrome image sensor may also have no color filters and therefore no color depth.

[0036] In some cases, the image sensor 230 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). The image sensor 230 may also include an analog gain amplifier to amplify the analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms 220 may alternatively or additionally be included in the image sensor 230. The image sensor 230 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a CCD / CMOS hybrid sensor (e.g., sCMOS), or other combinations.

[0037] The image processor 250 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 254), one or more host processors (including host processor 252), and / or one or more of any other types of processors 910 discussed with respect to the computing system 900. The host processor 252 may be a digital signal processor (DSP) and / or other types of processors. In some embodiments, the image processor 250 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 252 and the ISP 254. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 256), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 256 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an integrated circuit 2 (I2C) interface, an integrated circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a mobile industry processor interface (MIPI) (e.g., a MIPI CSI-2 physical (PHY) layer port or interface, an advanced high-performance bus (AHB) bus), any combination thereof, and / or other input / output ports. In one illustrative example, the main processor 252 may communicate with the image sensor 230 using an I2C port, and the ISP 254 may communicate with the image sensor 230 using a MIPI port.

[0038] The image processor 250 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging images to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 250 may store image frames and / or processed images in a random access memory (RAM) 140 / 3225, a read-only memory (ROM) 145 / 920, a cache, a storage unit, other storage devices, or some combination thereof.

[0039] Various input / output (I / O) devices 260 may be connected to the image processor 250. The I / O devices 260 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 935, any other input device 945, or some combination thereof. In some cases, captions may be entered into the image processing device 205B via a physical keyboard or keypad of the I / O device 260, or via a virtual keyboard or keypad of the touch screen of the I / O device 260. The I / O 260 may include one or more ports, jacks, or other connectors to enable wired connections between the image capture and processing system 200 and one or more peripheral devices, over which the image capture and processing system 200 may receive data from and / or send data to the one or more peripheral devices. The I / O 260 may include one or more wireless transceivers to enable wireless connections between the image capture and processing system 200 and one or more peripheral devices, over which the image capture and processing system 200 may receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 260 discussed previously, which themselves may be considered I / O devices 260 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0040] In some cases, the image capture and processing system 200 can be a single device. In some cases, the image capture and processing system 200 can be two or more separate devices, including an image capture device 205A (e.g., a camera) and an image processing device 205B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 205A and the image processing device 205B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 205A and the image processing device 205B can be disconnected from each other.

[0041] like Figure 2 As shown, a vertical dotted line will Figure 2 2 is a schematic diagram of an image capture and processing system 200. The image capture and processing system 200 is divided into two parts, and represents an image capture device 205A and an image processing device 205B. The image capture device 205A includes a lens 215, a control mechanism 220, and an image sensor 230. The image processing device 205B includes an image processor 250 (including an ISP 254 and a main processor 252), a RAM 240, a ROM 245, and an I / O 260. In some cases, some components shown in the image capture device 205A, such as the ISP 254 and / or the main processor 252, can be included in the image capture device 205A.

[0042] The image capture and processing system 200 may include an electronic device, such as a handheld mobile or fixed telephone (e.g., a smartphone, a cellular phone, etc.), a desktop computer, a notebook or laptop computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 200 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some embodiments, the image capture device 205A and the image processing device 205B may be different devices. For example, the image capture device 205A may include a camera device, while the image processing device 205B may include a computing device, such as a mobile phone, a desktop computer, or other computing device.

[0043] Although the image capture and processing system 200 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 200 may include more than Figure 2. The components of the image capture and processing system 200 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, the components of the image capture and processing system 200 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, GPU, DSP, CPU, and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and may be executed by one or more processors of an electronic device that implements the image capture and processing system 200. In some examples, the image adjustment system 300 may include the image capture and processing system 200, the image capture device 205A, the image processing device 205B, or a combination thereof.

[0044] Figure 3 is a block diagram illustrating an example of an image adjustment system 300. Image adjustment system 300 includes various components for processing one or more images, such as removing unwanted features or objects from one or more images. Image adjustment system 300 can remove the unwanted features or objects and replace the color values ​​of pixels at the locations of the removed objects with colors representing the portion of the captured scene in the one or more images that was occluded by the removed objects. As shown, components of image adjustment system 300 include one or more image capture devices 302, an object recognition engine 304, a feature segmentation engine 306, and an image adjustment engine 308.

[0045] In the following Figure 3 and the description of the image adjustment system 300, with reference to Figures 4A to 4F The example images shown in illustrate different aspects of image adjustment performed by the image adjustment system 300. Figures 4A to 4F In the example illustrated in FIG, person 402 is selected for adjustment (e.g., removal and / or replacement). Figure 4A The adjustment applied to the person 402 in FIG. 4 is to remove the person 402 from any captured image after identifying the person 402 as an unwanted object and replace the person with a background object 404 that is occluded by the person 402 .

[0046] refer to Figure 3, the image adjustment system 300 may include or be part of a handheld mobile or landline phone (e.g., a smartphone, cellular phone, or similar device), a server computer (e.g., in communication with a vehicle computing system), a desktop computer, a notebook or laptop computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the image adjustment system 300 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications, such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications. In some embodiments, components of the image adjustment system 300 (e.g., one or more image capture devices 302, object recognition engines, feature segmentation engines 306, and image adjustment engines 308) may be part of the same computing device. In some embodiments, components of the image adjustment system 300 may be part of two or more separate computing devices. In some cases, the image adjustment system 300 may be used as Figure 9 900 as shown.

[0047] Although the image adjustment system 300 is shown as including certain components, one of ordinary skill will appreciate that the image adjustment system 300 may include more than Figure 3 In some cases, the additional components of the image adjustment system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the image adjustment system 300 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 3 One or more other software and / or hardware components not shown in the figure. In some embodiments, the additional components of the image adjustment system 300 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a digital signal processor (DSP), a microprocessor, a microcontroller, a graphics processing unit (GPU), a central processing unit (CPU), any combination thereof, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executed by one or more processors of the electronic device implementing the image adjustment system 300.

[0048] One or more images can be captured by one or more image capture devices 302. One or more image capture devices 302 (e.g., cameras or other image sensors) can be included in a mobile device and can be oriented toward a user of the device (e.g., using one or more front-facing cameras) or away from the user of the device (e.g., using one or more rear-facing cameras).

[0049] Each of the one or more image capture devices 302 can include a camera or other type of image sensor. In some examples, one or more image capture devices 302 can include an infrared (IR) camera configured to capture IR images and / or near-infrared (NIR) images. For example, an IR camera or sensor can capture IR signals. The wavelength and frequency of IR signals fall within the IR electromagnetic spectrum. The IR electromagnetic spectrum has a wavelength range of 2,500 nanometers (nm) to 1 millimeter (mm), corresponding to a frequency range of 430 terahertz (THz) to 400 gigahertz (GHz). The infrared spectrum includes the NIR spectrum, which has a wavelength range of 780nm to 2,500nm. In some cases, the image adjustment system 300 can include an IR sensor configured to capture IR and NIR signals. In some cases, the image adjustment system 300 can include separate IR and NIR sensors. In some examples, one or more image capture devices 302 can include a camera configured to capture color images and / or monochrome images. Color images can include: red-green-blue (RGB) images; luminance, chrominance-blue, chrominance-red (YCbCr or Y'CbCr) images; and / or any other suitable type of image. In one illustrative example, image adjustment system 300 can include an RGB camera or multiple RGB cameras. In some cases, one or more image capture devices 302 can include one or more IR cameras and one or more RGB cameras.

[0050] In some embodiments, the one or more image capture devices 302 may include one or more depth sensors. The one or more depth sensors may obtain measurements of distances corresponding to objects in the captured scene. In one illustrative example, the depth sensor may take the form of a light source that projects a structured or textured light pattern (which may include one or more narrow light bands) onto one or more objects in the scene. By exploiting the geometric deformation of the projected pattern caused by the surface shape of the object, depth information may be obtained. In some cases, the one or more depth sensors may generate a depth image that includes depth values ​​corresponding to pixel locations in one or more images captured by the one or more image capture devices 302. In some cases, the depth sensor may be in the same approximate location as other sensors of the one or more image capture devices 302. In some cases, the depth sensor may capture the depth image at the same time as images are captured by one or more other sensors included in the one or more image capture devices 302. In some embodiments, the systems and techniques described herein may also be used when inferring depth information from one or more images.

[0051] In one illustrative example, the one or more image capture devices 302, and in some cases, one or more depth sensors, can capture one or more RGB-D images. In some cases, the one or more image capture devices 302 can capture other image types that include depth information, such as monochrome depth, NIR depth, etc. For ease of illustration, the examples of this disclosure discuss performing image adjustments on RGB-D images, but the systems and techniques described herein can also be used with other image types that include depth information without departing from the scope of this disclosure.

[0052] Figure 4A An example first input image 410 of a scene is shown. In the example shown, first input image 410 includes a person 402 in front of background objects 404. For example, background objects 404 include a tree and several tents. For the purpose of this example, an image adjustment process will be described for removing person 402 from an image of the scene while maintaining the accurate appearance of background objects 404.

[0053] Back to Figure 3 , one or more images captured by one or more image capture devices 302 may be provided as input to object recognition engine 304. In one illustrative example, Figure 4A The first input image 410 may be a first input image captured by an image capture device and displayed on a display. In some cases, the first input image may be displayed to provide a user with a preview of an image that will be captured after the image capture device receives a capture input (e.g., when a user presses a capture button). Figure 4B In the example, with Figure 4A An identification 406 corresponding to the positioning of person 402 in the image can be obtained by object recognition engine 304. For example, object recognition engine 304 can obtain an indication of one or more selected objects (or ROIs) in the input image to be adjusted. In some cases, the indication of one or more selected objects in the input image to be adjusted can be based on user input. In one illustrative example, the user may be able to indicate the one or more selected objects using physical contact (e.g., tapping a screen, swiping, etc.), a gesture (e.g., detected by a camera or other sensor), using an input device, or by any other means that allows the user to interact with image adjustment system 300.

[0054] In some cases, the pixel locations of the identified objects can be provided as input to the feature segmentation engine 306. For illustrative purposes, the feature segmentation engine 306 based on a machine learning model (e.g., a deep learning neural network) will be described below. However, it should be understood that feature segmentation based on other technologies, such as computer vision, digital image processing, thresholding, region-based segmentation, edge segmentation, Otsu's algorithm, clustering algorithm, any other feature segmentation technology or any combination thereof, can be used without departing from the scope of this disclosure. In some cases, the feature segmentation engine 306 can be trained to detect spatial information (e.g., features) associated with one or more images. In some cases, the feature segmentation engine 306 can be further trained to provide one or more classifications for objects in one or more images based on the detected features. The feature segmentation engine 306 can in turn use these classifications to segment the images into different parts associated with one or more classifications. For example, the feature segmentation engine 306 can segment one or more images into different parts associated with people, buildings, cars, furniture, plants, etc.

[0055] In some examples, the feature segmentation engine 306 can also be trained to classify objects corresponding to features extracted from one or more images using one or more classifications. A training dataset comprising example images and classification labels can be used to train the feature segmentation engine 306, such as using the following method: Figure 7 and Figure 8 In one illustrative example, during inference (e.g., after the feature segmentation engine 306 has been trained), the feature segmentation engine 306 can use the feature vectors to classify one or more input images. Example classifications can include "person," "face," "building," "tree," and any other classifications that the feature segmentation engine 306 has been trained to classify.

[0056] In one illustrative example, the feature segmentation engine 306 can perform semantic segmentation of the image. In semantic segmentation, the feature segmentation engine 306 can associate each object in the scene with one or more classifications (also referred to herein as labels). In some cases, if more than one object has the same label, the semantic segmentation will not distinguish between the two objects. In such an example, depth information in one or more captured images (e.g., RGB-D images) can be used to distinguish between objects and further segment between objects. For example, a portion of the input image containing two people can be classified using a single "person" classifier. Figure 4B An example image 420 is shown depicting a classifier 408 (shown in white) associated with the person 402 in the first input image 410. In some examples, the classifier 408 may represent a classification such as "person" or "face." In some aspects, through semantic segmentation, the feature segmentation engine 306 may not be able to individually identify overlapping objects as different objects.

[0057] In another illustrative example, the feature segmentation engine 306 can perform instance segmentation. In instance segmentation, the feature segmentation engine 306 can use the same classifier to separately and differently identify multiple instances of an object. For example, in a single image such as image 100, where there are multiple people, instance segmentation can be used to distinguish between five people (e.g., two people 102 and people 104A, 104B, 104C) by assigning a separate classifier to each identified person. As a result, a subset of people can be selected for removal (e.g., people 104A, 104B, 104C), while a different subset of people (e.g., person 102) can be retained in the final adjusted image.

[0058] In some cases, based on features, objects, and / or categories determined from the first input image, the object recognition engine 304, the feature segmentation engine 306, and / or the image adjustment engine 308 may convert the input image (e.g., Figure 4A The first input image 410, Figure 4D Pixel locations in the second input image 440) are associated with corresponding features, objects, and / or classifications determined from the input image by the feature segmentation engine 306. In some cases, the pixel locations associated with the unwanted object(s) can be provided as input to the image adjustment engine 308. In some cases, the features, objects, and / or classifications associated with the unwanted object(s) can be stored in a memory (not shown) and used to identify the unwanted objects in subsequent images.

[0059] Once the image adjustment engine 308 obtains the pixel locations associated with the unwanted object(s) from the object recognition engine 304 and / or the feature segmentation engine 306, the image adjustment engine 308 can adjust the pixel locations associated with the unwanted object(s) in each image. In some cases, based on the identified pixel locations, the image adjustment engine 308 can remove and / or replace the unwanted objects to form an adjusted image.

[0060] In some cases, the image adjustment engine 308 may perform an inpainting process to replace pixels in the input image that were previously occupied by the unwanted object. In some cases, the image adjustment engine 308 may obtain one or more additional images of the same scene or portion of the scene as the first input image. In some cases, pixels occluded by the unwanted object(s) (e.g., person 402) in the first input image may not be occluded in another image of the scene or portion of the scene processed by the image adjustment system 300. In some cases, unoccluded pixel information corresponding to the location of the unwanted object in the first input image may be used to replace the unwanted object in the first adjusted image. For example, pixels from two or more images may be combined to generate the adjusted image (e.g., image fusion). Figure 4C A first adjusted image 430 is shown in which an unwanted object (eg, person 402) has been removed and replaced (eg, in Figure 4C Position 412).

[0061] In some cases, once unwanted objects are identified, classified, and / or removed by the image adjustment system 300, features, objects, and / or classifications associated with the unwanted object(s) may be retrieved from storage (not shown) and used to identify the unwanted object(s) in subsequent images.

[0062] Figure 4D A second input image 440 is shown, which also includes Figure 4A In some cases, the feature segmentation engine 306 can determine a classifier 408 for the person 402 from the second image 440. Accordingly, Figure 4EThe image 450 shown in FIG. 4 shows the identification 416 of the new location of the person 402 in the second input image. In some cases, the object recognition engine 304 can obtain the classification generated by the feature segmentation engine 306 and determine that the identified object match is a common object that is present in both the first input image 410 and the second input image 440. Based on the identification of the unwanted object in the second input image 440 by the object recognition engine 304, the image adjustment engine 308 can remove and / or replace the person 402 from the background object 404 to generate the image shown in FIG. Figure 4F The second adjusted image 460 is shown. Figure 4F In the example shown, replaced pixels 418 may correspond to correspondingly located unoccluded pixels in first input image 410 and / or first adjusted image 430, as indicated by white outline 414. As additional images are acquired from one or more image capture devices 302, image adjustment system 300 may track identified unwanted objects in a similar manner and remove and / or replace the unwanted objects.

[0063] Although the examples described herein are provided with respect to identifying and removing unwanted objects, in some cases, target objects that are desired to be included in the adjusted image can be identified, and the image adjustment system 300 can determine whether there are any objects in the scene that have not been identified as target objects. For example, the object recognition engine 304 and / or the feature segmentation engine 306 can determine whether objects in the scene that have not been identified as target objects have moved, and identify the moved objects as objects to be removed. In another example technique, depth information captured by one or more image capture devices 302 can be used to identify unwanted objects. For example, if the depth of the target object exceeds the depth of another detected object (e.g., an object detected by the object recognition engine 304) by a threshold amount, the other detected object can be identified as an unwanted object. Regardless of the technique used to identify the target object and / or unwanted objects, the systems and techniques described herein can be used to remove and / or replace unwanted object(s) from the adjusted image.

[0064] In some cases, the object recognition engine 304 may obtain additional input to guide the identification of objects as target objects and / or unwanted objects. For example, the object recognition engine 304 may obtain user input and / or audio input to guide object recognition. In one illustrative example, user input may include identifying only family members, friends, and / or pets. For example, the object recognition engine 304 may be trained to recognize people, places, pets, etc. in a photo album stored on a storage device (not shown). In another illustrative example, the object recognition engine 304 may obtain user gaze information to determine a region of interest corresponding to the user's gaze. In some embodiments, the object recognition engine 304 may determine that objects outside the region of interest are unwanted objects. In some cases, the object recognition engine 304 may identify target objects within the region of interest. In another illustrative example, the object recognition engine 304 may obtain audio input corresponding to the input image to determine target objects and / or unwanted objects. For example, the object recognition engine 304 may determine that captured audio corresponds to a family member, friend, pet, etc., and determine a region of interest in the captured image corresponding to the source of the captured audio. In another illustrative example, object recognition engine 304 may determine that the captured audio corresponds to an unrecognized person or pet and identify one or more unwanted objects in the captured image that correspond to the source of the captured audio.

[0065] In the example shown, Figure 3 The object recognition engine 304 and the feature segmentation engine 306 are shown separately. In some cases, identifying one or more removal objects and / or one or more target objects can be performed after the input image is segmented by the feature segmentation engine 306. For example, the feature segmentation engine 306 can be trained to identify people, places, pets, etc. in the input image as target objects. For example, the feature segmentation engine 306 can be trained to identify features in the input image that correspond to previously captured people, places, pets, etc. In some cases, the object recognition engine can take the features determined by the feature segmentation engine 306 as input and can determine unwanted objects in the scene captured in the input image based on other factors, such as the movement of objects between subsequent images.

[0066] In some cases, the image adjustment system 300 may obtain Figure 4A The first input image 410 in Figure 4D440 in the second input image 440). For example, the electronic device may include a first image capture device and a second image capture device (e.g., included in one or more image capture devices 302), which simultaneously capture at least a portion of a common portion of the scene. In some cases, the object recognition engine 304 may determine a target object and / or unwanted objects based on the image obtained from the first image capture device, and remove and / or replace the unwanted objects from the image obtained from the second image capture device. In one illustrative example, the first image capture device may have a lower resolution and / or lower power consumption than the second image capture device. For example, the first image capture device may include an always-on (AON) camera. In some cases, a single image sensor may capture multiple images of a scene to provide, for example, zero shutter lag (ZSL) capture, preview images, etc.

[0067] Figure 5 5 is a block diagram illustrating an image capture system 500. As shown, the image capture system 500 includes a ZSL buffer 532, an image processing engine 540, an image adjustment engine 545, and a preview buffer 536. The image adjustment engine 545 can be used with Figure 3 5. The image adjustment system 500 corresponds to the image adjustment system 300 shown in FIG. 5. It should be understood that the image capture system 500 is an example block diagram provided for illustrative purposes and may include more or fewer components without departing from the scope of the present disclosure. Furthermore, although the image adjustment engine 545 is shown as being included in the image processing engine 540, the image adjustment engine 545 may be independent of the image processing engine 540 without departing from the scope of the present disclosure.

[0068] like Figure 5 As shown, a timeline 501 of an example image capture sequence includes a capture event 503 (eg, capture input). In the example shown, an image may be captured from an image sensor 530 (eg, Figure 2 Image sensor 230, Figure 3 One or more image capture devices 302, etc., capture images during the preview and ZSL period 502 before the capture event 503. In addition, after the capture event 503, the image capture period 504 may include capturing images after the capture event 503. Figure 5 In the example shown, image capture system 500 can be configured to provide a preview image with unwanted objects removed from the preview image (e.g., before receiving a capture input). In some cases, image capture system 500 can provide a preview image before, during, and / or after capture event 503.

[0069] In some embodiments, the image capture system 500 can obtain images from the image sensor 530, which can be connected to the image sensor 530. Figure 3 302 in the image capture device 300. As shown, a zero shutter lag (ZSL) buffer 532 can be used to store images or frames captured by the image sensor 530. In some examples, the ZSL buffer 532 is a circular buffer. Generally speaking, the ZSL buffer 532 can be used to store one or more frames recently captured by the sensor, which can compensate for lag time that may occur until the image capture system 500 completes encoding and storing the frame in response to receiving a shutter (or capture) command (e.g., based on user input or automatically received). The ZSL buffer 532 can be coupled to the image processing engine 540 to store the image before the image is output and stored to a storage device (e.g., Figure 9 For example, the image processing engine 540 may perform image processing on the image in response to the capture event 503.

[0070] In some cases, the image processing engine 540 can generate a preview image and output the preview image to the preview buffer 536. For example, the image processing engine 540 can generate the preview image by scaling and / or cropping the raw image data from the image sensor 530 to match the size of the preview display (e.g., the display of the electronic device). In some cases, as described above, the image adjustment engine 545 can identify unwanted objects in the preview image (e.g., based on user input or automatic determination). In some cases, once the unwanted objects are identified, the image adjustment engine 545 can remove and / or replace the unwanted objects from the preview image and provide the adjusted preview image to the preview buffer 536. As a result, the image output to the preview display 550 can provide an indication of how the captured image would look after the unwanted objects are removed. In some cases, the image adjustment engine 545 can utilize an inpainting process during the preview and ZSL period 502 to replace pixels associated with the unwanted objects. For example, an inpainting process using interpolation can be used to avoid delays associated with combining pixels from multiple images. In some cases, segmentation can be performed on the scaled and / or cropped preview image (e.g., by Figure 3 536) to determine the location of the unwanted object(s). In some cases, each preview image stored in preview buffer 536 can be indexed to a corresponding image stored in ZSL buffer 532. In some cases, the segmentation provided by feature segmentation engine 306 can be stored in preview buffer 536 in addition to or instead of the preview images.

[0071] In some cases, after capture event 503 occurs, image processing engine 540 may retrieve images from ZSL buffer 532 for processing and output to storage (referred to herein as processed images). In some cases, processing of the processed images may occur during image processing period 506. As described above, each image in ZSL buffer 532 may be indexed to a corresponding preview image and / or segmentation in preview buffer 536. In some cases, the segmentation identified as a preview image may be reused by image adjustment engine 545 to identify pixels in the processed image corresponding to the undesired object(s). In some cases, when the undesired object(s) are located at different locations in the scene in the images stored in ZSL buffer 532, pixels in the processed image that are occluded by the undesired object(s) may be replaced with corresponding pixels from other images in ZSL buffer 532 that are not occluded by the undesired object(s) (e.g., at the location of the occluded pixels). However, in some cases, all images stored in the ZSL buffer 532 may not include unobstructed views of some or all of the pixels that were occluded in the processed image. For example, if the unwanted object remains stationary for a long period of time, the ZSL buffer may not include any images without the unwanted object. In this case, the image adjustment engine 545 may utilize the methods described above with respect to Figure 3 In some cases, the image adjustment engine 545 may combine pixels from multiple images (e.g., image fusion), perform inpainting, perform any other technique for replacing pixels of an image, and / or any combination thereof.

[0072] In one illustrative example, the image capture system 500 can be configured to use inpainting to remove unwanted objects during the preview and ZSL period 502. In some examples, the image capture system 500 can combine pixels from multiple images for the unwanted object(s) in the ZSL buffer 532 that have moved during the capture time period, and perform inpainting for any other pixels that are occluded by the unwanted object(s). However, any combination of pixel replacement techniques can be used during the preview and ZSL period 502, the image capture period 504, the image processing period 506, or any other operating period of the image capture device without departing from the scope of the present disclosure.

[0073] As described above, the image adjustment system 300 and related techniques described herein can be utilized to remove unwanted objects from images captured by an image capture device (e.g., a camera). Furthermore, using the techniques described herein, a photographer can be provided with a preview image displayed on a display that reflects the scene after the unwanted object(s) have been removed. In some cases, the unwanted objects can also be automatically identified (e.g., by a machine learning model) and removed from the captured image, thereby avoiding the need for post-processing. Furthermore, because the systems and techniques described herein can be used to remove unwanted objects from images stored in an image buffer (e.g., a Figure 5 532) is combined so that an accurate reproduction of the captured scene without unwanted object(s) can be generated by the image adjustment systems and techniques. In some cases, to conserve system resources, other object removal and replacement techniques, such as inpainting, can be used to conserve system resources. For example, inpainting techniques can be used during a preview period (e.g., before capture input is received). In some cases, the preview image may not ultimately be stored on a storage device, and image quality can be sacrificed in favor of low latency so that the preview image is more likely to match the processed capture image after the capture input is received.

[0074] Figure 6 6 is a flow chart illustrating an example of a process 600 for processing one or more images. At block 602, the process 600 includes acquiring a first image of a scene from a camera (e.g., one or more image capture devices 302). In some cases, the scene includes a first object at a first location and a second object at a second location (see FIG. Figure 4B ).

[0075] At block 604, the process 600 includes acquiring a second image of the scene from the camera. In some examples, the second image of the scene includes a first object at a first location and a second object at a third location, the third location being different from the second location (see FIG. Figure 4E In some cases, the first image of the scene and the second image of the scene are acquired before the capture input is acquired (e.g., stored in Figure 5 ) preview image in preview buffer 536. In some examples, the second image of the scene is an image acquired after the capture input is acquired.

[0076] At block 606, the process 600 includes generating an adjusted second image based on the second image (e.g., by the image adjustment engine 308). In some cases, the adjusted second image includes the first object at the first location, and the second object at the third location is removed from the adjusted second image (see Figure 4FIn some cases, removing the second object from the adjusted second image includes interpolating pixel colors of the second image to generate pixel data for one or more pixels in the second image of the scene associated with the second object. In some examples, removing the second object from the adjusted second image includes obtaining pixels of the first image of the scene associated with a third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0077] At block 608, the process 600 includes displaying a Figure 5 The adjusted second image is displayed on a preview display 550 of the scene. In some examples, the first image of the scene and the adjusted second image include images captured independently of obtaining the capture input (e.g., the preview image).

[0078] In some examples, the process 600 includes acquiring a third image of the scene from a camera, acquiring a capture input, and generating a captured image based on the third image of the scene based on acquiring the capture input.

[0079] In some embodiments, the process 600 includes, after displaying the first image of the scene on the display, obtaining input associated with selecting a second object to be removed from the scene. In some examples, the input associated with selecting the second object to be removed from the scene includes at least one or more of physical contact, a gesture, a gaze direction, an input from an input device, or an audio input. In some examples, the process 600 includes obtaining input associated with selecting the first object to be included as a target object in the adjusted second image. In some embodiments, the input associated with selecting the first object to be included as a target object in the adjusted second image includes at least one or more of physical contact, a gesture, a gaze direction, an input from an input device, or an audio input.

[0080] In some cases, the process 600 includes determining that the second object is an unwanted object (e.g., by Figure 3 In some cases, determining that the second object is an unwanted object includes at least one or more of determining that the second object moved between the first image of the scene and the second image of the scene, determining that the first object is a previously captured object and the second object is not a previously captured object, determining that a depth of the first object differs from a depth of the second object by an amount greater than a threshold depth, determining that the second object is outside of the region of interest based on a gaze direction, or determining that audio input associated with the second object indicates an unwanted object.

[0081] In some cases, the process 600 includes acquiring a capture input. In some examples, the process 600 includes acquiring a third image of the scene. In some aspects, the third image of the scene includes the first object at the first location and the second object. In some cases, the process 600 includes determining a fourth location of the second object in the scene. In some embodiments, the process 600 includes generating an adjusted third image that includes the first object at the first location. In some cases, removing the second object from the adjusted third image. In some examples, the process 600 includes storing the adjusted third image. In some embodiments, the process 600 includes acquiring a segmentation associated with the second object and the first image of the scene (e.g., by Figure 3 In some examples, the first image has a first resolution (e.g., Figure 5 The preview image in the preview buffer 536 of FIG), and the third image has a second resolution (eg, Figure 5 In some examples, generating the adjusted second image includes performing an inpainting process to remove the second object from the second image of the scene, and generating the adjusted third image includes combining pixels of a fourth image of the scene associated with the fourth position with pixels of the third image of the scene. In some examples, the pixels of the fourth image of the scene associated with the fourth position include portions of the scene that were occluded by the second object in the third image of the scene.

[0082] In some cases, process 600 includes acquiring a fourth image of a scene from a camera, wherein the scene includes a first object at a first location and a second object at a second location. In some examples, the fourth image of the scene includes an image captured independently of acquiring a capture input. In some embodiments, process 600 includes generating an adjusted fourth image based on the fourth image. In some examples, the adjusted fourth image includes the first object at the first location, with the second object removed from the adjusted fourth image. In some aspects, process 600 includes displaying the adjusted fourth image on a display. In some examples, process 600 includes acquiring an input associated with removing the second object based on the first image of the scene. In some embodiments, process 600 includes acquiring a fifth image of the scene from a camera, acquiring a capture input, and generating a captured image based on the fifth image of the scene based on acquiring the capture input. In some examples, the captured image includes the first object at the first location, with the second object at the second location removed from the captured image. In some embodiments, process 600 includes storing the captured image.

[0083] In some examples, the processes described herein (e.g., process 600 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, one or more processes can be performed by Figure 3 In another example, one or more processes may be performed by the image adjustment system 300. Figure 9 The computing system 900 shown in FIG. Figure 9 The computing device of the illustrated computing system 900 may include components of the image adjustment system 300 and may implement Figure 6 The operations of process 600 and / or other processes described herein.

[0084] The computing device may include any suitable device, such as a vehicle or a computing device of a vehicle (e.g., a driver monitoring system (DMS) of a vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a watch or smartwatch connected to a network, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein, including process 600 and / or other processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0085] Components of a computing device may be implemented in circuitry. For example, the component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0086] Process 600 is illustrated in the form of a logical flow diagram, the operations of which represent a sequence of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order of the operations described is not intended to be construed as limiting, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0087] Additionally, the process 600 and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is collectively executed on one or more processors through hardware or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0088] As described above, various aspects of the present disclosure may utilize machine learning models or systems. Figure 7 is an illustrative example of a deep learning neural network 700 that can be used to implement the above-described machine learning-based feature segmentation, instance segmentation, depth estimation, and / or classification. Input layer 720 includes input data. In one illustrative example, input layer 720 may include data representing pixels of an input image. Neural network 700 includes multiple hidden layers 722a, 722b, to 722n. Hidden layers 722a, 722b, to 722n include "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers may be determined based on a given application to include the number of hidden layers required for the given application. Neural network 700 also includes an output layer 721 that provides an output resulting from the processing performed by hidden layers 722a, 722b, to 722n. In one illustrative example, output layer 721 may be an input image (e.g., Figure 4A The first input image 410 and Figure 4D The object in the second input image 440 is provided with classification. The classification may include identifying the category of the activity type (e.g., raising the head, lowering the head, closing the eyes, yawning, etc.).

[0089] Neural network 700 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. The information associated with a node is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 700 can include a feedforward network, in which case there are no feedback connections in which the output of the network is fed back to itself. In some cases, neural network 700 can include a recurrent neural network, which can have loops that allow information to be passed between nodes while reading input.

[0090] Information can be exchanged between nodes in each layer through node-to-node connections. Nodes in input layer 720 can activate a set of nodes in first hidden layer 722a. For example, as shown, each input node in input layer 720 is connected to each node in first hidden layer 722a. The nodes in first hidden layer 722a can transform the information of each input node by applying an activation function to the input node information. The resulting information can then be passed to nodes in the next hidden layer 722b and activated, which can then perform their own designated functions. Example functions include convolution, upsampling, data conversion, and / or any other suitable function. The output of hidden layer 722b can activate nodes in the next hidden layer, and so on. The output of the last hidden layer 722n can activate one or more nodes in output layer 721, which then provides an output. In some cases, although nodes in neural network 700 (e.g., node 726) are shown as having multiple output lines, a node has only a single output, and all lines shown as outputs from a node represent the same output value.

[0091] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters trained by neural network 700. Once neural network 700 is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, the interconnections between nodes can represent a piece of learned information about the interconnected nodes. The interconnections can have adjustable numerical weights that can be adjusted (e.g., based on a training data set), allowing neural network 700 to adapt to the input and learn as it processes more and more data.

[0092] Neural network 700 is pre-trained to process features from data in input layer 720 using various hidden layers 722a, 722b, through 722n to provide output via output layer 721. In examples where neural network 700 is used to identify features in images, neural network 700 can be trained using training data that includes both images and labels, as described above. For example, training images can be input to the network, each with a label indicating the features in the image (for a feature segmentation machine learning system) or a label indicating the class of activity in each image. In an example illustrated using object classification, the training images can include images of the number 2, in which case the labels for the images can be [0 0 1 0 0 0 0 0 0].

[0093] In some cases, the neural network 700 can use a training process called backpropagation to adjust the weights of the nodes. As described above, the backpropagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed in one training iteration. This process can be repeated a certain number of times for each set of training images until the neural network 700 is trained well enough so that the weights of each layer are accurately adjusted.

[0094] For the example of recognizing objects in an image, the forward pass can include passing a training image through the neural network 700. Prior to training the neural network 700, the weights are initially random. As an illustrative example, the image can include an array of numbers representing the pixels of the image. Each number in the array can include a value from 0 to 255 that describes the intensity of the pixel at that position in the array. In one example, the array can include a 28x28x3 array of numbers with 28 rows and 28 columns of pixels and three color components (such as red, green, and blue, or brightness and two chrominance components, etc.).

[0095] As described above, for the first training iteration of the neural network 700, since the weights are randomly selected at initialization, the output is likely to include values ​​that are not biased towards any particular class. For example, if the output is a vector with probabilities that the object includes different classes, the probability values ​​for each different class may be equal or at least very similar (e.g., for ten possible classes, the probability value for each class may be 0.1). With the initial weights, the neural network 700 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function can be used to analyze the error in the output. Any suitable loss function definition can be used, such as cross entropy loss. Another example of a loss function includes the mean squared error (MSE), which is defined as The loss can be set equal to the value E total .

[0096] The loss (or error) for the first training image will be large because the actual value is very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 700 can perform a backward pass by determining which inputs (weights) contribute the most to the network loss, and the weights can be adjusted so that the loss is reduced and ultimately minimized. The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes the most to the network loss. After the derivative is calculated, the weight update can be performed by updating all the weights of the filter. For example, the weight can be updated so that it changes in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Denotes the initial weights, and η denotes the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates and a low value indicates smaller weight updates.

[0097] The neural network 700 may include any suitable deep network. One example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 700 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), and others.

[0098] Figure 8 is an illustrative example of a convolutional neural network (CNN) 800. The input layer 820 of the CNN 800 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, with each number in the array including a value from 0 to 255 that describes the intensity of the pixel at that position in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luma and two chroma components, etc.). The image may be passed through a convolutional hidden layer 822a, an optional nonlinear activation layer, a pooling hidden layer 822b, and a fully connected hidden layer 822c to obtain an output at an output layer 824. Although Figure 8 Only one of each hidden layer is shown, but one of ordinary skill will understand that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 800. As previously described, the output may indicate a single class of object, or may include a probability of a class that best describes the object in the image.

[0099] The first layer of CNN 800 is a convolutional hidden layer 822a. Convolutional hidden layer 822a analyzes the image data from input layer 820. Each node in convolutional hidden layer 822a is connected to a region of nodes (pixels) in the input image, which is called a receptive field. Convolutional hidden layer 822a can be viewed as one or more filters (each filter corresponding to a different activation or feature map), where each convolution iteration of the filter is a node or neuron in convolutional hidden layer 822a. For example, the region of the input image covered by the filter in each convolution iteration will be the receptive field of the filter. In one illustrative example, if the input image comprises a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in convolutional hidden layer 822a. Each connection between a node and its receptive field learns a weight, and in some cases, an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node in the hidden layer 822a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has the same depth and weight (number) array as the input. Taking an image frame as an example (based on the three color components of the input image), the depth of the filter will be 3. An illustrative example size of the filter array is 5x5x3, corresponding to the size of the node's receptive field.

[0100] The convolutional nature of the convolutional hidden layer 822a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 822a can start from the upper left corner of the input image array and can be convolved around the input image. As described above, each convolution iteration of the filter can be regarded as a node or neuron of the convolutional hidden layer 822a. In each convolution iteration, the filter value is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications of each convolution iteration can be added to obtain the sum for that iteration or node. Next, this process is continued at the next location in the input image based on the receptive field of the next node in the convolutional hidden layer 822a. For example, the filter can be moved to the next receptive field by a step size (called a stride). The stride can be set to 1 or other suitable value. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thereby determining a sum value for each node in the convolutional hidden layer 822a.

[0101] The mapping from the input layer to the convolutional hidden layer 822a is called an activation map (or feature map). The activation map includes the value of each node, which represents the filter result at each location of the input volume. The activation map may include an array that includes the various summed values ​​produced by each iteration of the filter on the input volume. For example, if a 5x5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24x24 array. The convolutional hidden layer 822a may include multiple activation maps to identify multiple features in the image. Figure 8 The example shown in includes three activation maps. Using three activation maps, the convolutional hidden layer 822a can detect three different features, each of which can be detected across the entire image.

[0102] In some examples, a nonlinear hidden layer can be applied after the convolutional hidden layer 822a. A nonlinear layer can be used to introduce nonlinearity to a system that has been computing linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. The ReLU can thus increase the nonlinear properties of the CNN 800 without affecting the receptive field of the convolutional hidden layer 822a.

[0103] The pooling hidden layer 822b can be applied after the convolutional hidden layer 822a (and after it when a non-linear hidden layer is used). The pooling hidden layer 822b is used to simplify the information in the output from the convolutional hidden layer 822a. For example, the pooling hidden layer 822b can obtain each activation map output from the convolutional hidden layer 822a and use a pooling function to generate a compressed activation map (or feature map). Max pooling is an example of a function performed by a pooling hidden layer. Other forms of pooling functions are also used by the pooling hidden layer 822a, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 822a. In Figure 8 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 822a.

[0104] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the size of the filter, such as a stride of 2) to the activation map output by the convolutional hidden layer 822a. The output of the max pooling filter includes the maximum number in each sub-region within the convolution range of the filter. Using a 2×2 filter as an example, each unit in the pooling layer can summarize an area of ​​2×2 nodes in the previous layer (where each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum value from the four values ​​will be output as the "maximum" value. If such a max pooling filter is applied to the activation filter from the convolutional hidden layer 822a having a size of 24×24 nodes, the output from the pooling hidden layer 822b will be an array of 12×12 nodes.

[0105] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves computing the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of computing the maximum value as in max pooling), and using the computed value as the output.

[0106] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image and discards the exact location information. Doing so may not affect the results of feature detection because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. The benefit provided by max pooling (and other pooling methods) is that it reduces the number of pooled features, thereby reducing the number of parameters required for subsequent layers of CNN 800.

[0107] The last layer of connections in this network is a fully connected layer that connects every node from the pooling hidden layer 822b to every output node in the output layer 824. Using the example above, the input layer includes 28×28 nodes that encode the pixel intensities of the input image; the convolutional hidden layer 822a includes 3×24×24 hidden feature nodes based on a 5×5 local receptive field (for the filter) applied to the three activation maps, and the pooling hidden layer 822b includes a layer of 3×12×12 hidden feature nodes based on a max pooling filter applied to a 2×2 region across each of the three feature maps. As an extension of this example, the output layer 824 can include 10 output nodes. In this example, every node of the 3×12×12 pooling hidden layer 822b is connected to every node of the output layer 824.

[0108] The fully connected layer 822c can take the output of the previous pooling hidden layer 822b (which should represent an activation map of high-level features) and determine the features that are most relevant to a particular category. For example, the fully connected layer 822c layer can determine the high-level features that are most relevant to a particular category and can include weights (nodes) for the high-level features. The product of the weights of the fully connected layer 822c and the weights of the pooling hidden layer 822b can be calculated to obtain the probabilities of different categories. For example, if the CNN 800 is used to predict that the object in the image is a person, there will be high values ​​in the activation map of high-level features representing people (e.g., the presence of two legs, a face at the top of the object, two eyes at the upper left and upper right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other common features of people).

[0109] In some examples, the output from the output layer 824 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of categories that the CNN 800 must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent the probability that the object belongs to a certain category. In one example, if a 10-dimensional output vector represents 10 different categories of objects as [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that the probability that the image is an object of the third category (e.g., a dog) is 5%, the probability that the image is an object of the fourth category (e.g., a person) is 80%, and the probability that the image is an object of the sixth category (e.g., a kangaroo) is 15%. The probability of a category can be regarded as the confidence that the object is part of that category.

[0110] Figure 9 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 9 An example of a computing system 900 is shown, which can be any computing device, such as one comprising an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 905. Connection 905 can be a physical connection using a bus, or a direct connection within processor 910, such as in a chipset architecture. Connection 905 can also be a virtual connection, a network connection, or a logical connection.

[0111] In some embodiments, computing system 900 is a distributed system, in which the functionality described in this disclosure can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the system components described represent a plurality of such components, each of which performs some or all of the functionality described for that component. In some embodiments, a component can be a physical device or a virtual device.

[0112] The example computing system 900 includes at least one processing unit (CPU or processor) 910 and connections 905 that couple various system components, including system memory 915, such as read-only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910. The computing system 900 may include a cache 912, which is a high-speed memory directly connected to the processor 910, close to the processor 910, or integrated as part of the processor 910.

[0113] Processor 910 may include any general-purpose processor and hardware or software services, such as services 932, 934, and 936 stored in storage device 930, configured to control processor 910 and a dedicated processor that incorporates software instructions into the actual processor design. Processor 910 is essentially a completely self-contained computing system, including multiple cores or processors, a bus, a memory controller, a cache, etc. Multi-core processors can be symmetric or asymmetric.

[0114] To enable user interaction, the computing system 900 includes an input device 945, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 900 can also include an output device 935, which can be one or more of a variety of output mechanisms. In some instances, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 900. The computing system 900 can include a communication interface 940, which generally hosts and manages user input and system output. The communication interface can use wired and / or wireless transceivers to perform or assist in receiving and / or sending wired or wireless communications, including using audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, temporary (ad-hoc) network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 900 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. Since there is no restriction on operating on any specific hardware arrangement, as hardware or firmware arrangements evolve, the basic features herein can be easily substituted for improved hardware or firmware arrangements.

[0115] The storage device 930 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flash memory card, a hard disk, a magnetic tape, a magnetic stripe, any other magnetic storage medium, flash memory, memristive memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0116] Storage device 930 may include software services, servers, services, etc. that, when the code defining such software is executed by processor 910, enable the system to perform functions. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium connected to the necessary hardware components, such as processor 910, connection 905, output device 935, etc., to perform that function.

[0117] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (multiple) instructions and / or data. Computer-readable media may include non-transitory media that can store data, which does not include carrier waves and / or temporary electronic signals that are propagated wirelessly or through a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, actual parameters, parameters, or memory contents. Information, actual numbers, parameters, data, etc. may be transmitted, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0118] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0119] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it is understood by those skilled in the art that the embodiments can be implemented without these specific details. For clear explanation, in some instances, the present technology may be described as including a single functional block, including a functional block that includes a device, a device component, a step or routine in a method implemented in software, or a combination of hardware and software. Additional components other than the components shown in the figures and / or the components described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams to avoid blurring the embodiments in unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may not show unnecessary details to avoid blurring the embodiments.

[0120] Individual embodiments may be described above as processes or methods, which may be described as flowcharts, flow charts, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates upon completion of its operations, but may also have additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, or the like. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or main function.

[0121] The processes and methods according to the above examples can be implemented using computer-executable instructions that are stored on or otherwise accessible from a computer-readable medium. For example, the instructions may include instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform certain functions or groups of functions. Some of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0122] The equipment according to these disclosure implementation processes and methods can include hardware, software, firmware, middleware, microcode, hardware description language or its arbitrary combination, and can adopt any one of various form factors.When being implemented in software, firmware, middleware or microcode, the program code or code segment (such as, computer program product) that performs the necessary tasks can be stored in computer-readable or machine-readable medium.(a plurality of) processors can perform the necessary tasks.Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. of notebook computers, smart phones, mobile phones, tablet devices or other small sizes.Functions described herein can also be implemented in peripheral or add-on cards.By further illustrating, these functions can also be implemented in different chips on circuit boards or in different processes running in single devices.

[0123] Instructions, media for conveying the instructions, computing resources for executing the instructions, and other structures for supporting the computing resources are example ways to provide the functions described in this disclosure.

[0124] In the above description, various aspects of the present application have been described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and used in various other ways, and the appended claims are intended to be interpreted as including these changes, unless limited by the prior art. The various features and aspects of the above application can be used alone or in combination. In addition, without departing from the scope of this specification, the embodiments can also be used in any environment and application other than those described herein. Therefore, this specification and the accompanying drawings should be regarded as illustrative rather than restrictive. For illustrative purposes, the method is described in a specific order. It should be understood that in alternative embodiments, the method can be performed in an order different from that described.

[0125] A person of ordinary skill understands that the less than ("<") and greater than (">") symbols or terms used in this document may be replaced by the less than or equal to ("<=") and greater than or equal to (">=") symbols, respectively, without departing from the scope of this description.

[0126] When a component is described as being “configured to” perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0127] The phrase “coupled to” means any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0128] Claim language or other language that recites "at least one" of a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" or "at least one of A or B" refers to A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" or "at least one of A, B, or C" refers to A, B, C, or A and B, or A and C, or B and C, or A, B, and C. Language that recites "at least one" of a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0129] The various illustrative logic blocks, modules, circuits, and algorithmic steps associated with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above from a functional perspective. Whether these functions are implemented in the form of hardware or software depends on the specific application and the design limitations of the entire system. Those skilled in the art can implement the functions in different ways with respect to each specific application, but the decision of this embodiment should not be interpreted as departing from the scope of this application.

[0130] The techniques described herein may also be implemented by electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices, which have multiple uses, including applications in wireless communication devices and other devices. Any functions described as modules or components may be implemented together in an integrated logic device, or may be implemented separately as separate but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium, which includes program code containing instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) of a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory (FLASH), a magnetic or optical data storage medium, and the like. Additionally or alternatively, these techniques may also be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0131] The program code can be run by a processor, which can include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. These processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor, but as an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Correspondingly, the term "processor" as used herein can refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0132] Illustrative aspects of the present disclosure include:

[0133] Aspect 1: An apparatus for processing one or more images. The apparatus includes a memory (e.g., implemented in a circuit) and a processor (or multiple processors) coupled to the memory. The processor (or multiple processors) is configured to: acquire a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position; acquire a second image of the scene from the camera, wherein the second image of the scene includes the first object located at the first position and the second object located at a third position, the third position being different from the second position; generate an adjusted second image based on the second image, wherein the adjusted second image includes the first object located at the first position, and the second object at the third position is removed from the adjusted second image; and display the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring a capture input.

[0134] Aspect 2: The apparatus of Aspect 1, wherein the processor is configured to: acquire a third image of the scene from the camera; acquire a capture input; and based on acquiring the capture input, generate a captured image based on the third image of the scene, wherein the captured image includes a first object located at the first position, and a second object located at the third position is removed from the captured image.

[0135] Aspect 3: The apparatus of any one of Aspects 1 to 2, wherein the second image of the scene and the third image of the scene are associated with a common identifier.

[0136] Aspect 4: The apparatus of any of Aspects 1 to 3, wherein the third image of the scene comprises an image acquired before the input is captured, wherein the third image of the scene is stored in at least one memory.

[0137] Aspect 5: The apparatus of any of Aspects 1 to 4, wherein the at least one memory comprises a zero shutter lag (ZSL) buffer.

[0138] Aspect 6: The apparatus of any one of Aspects 1 to 5, wherein the processor is configured to: obtain input associated with selecting a second object to be removed from the scene.

[0139] Aspect 7: The apparatus of any one of Aspects 1 to 6, wherein the processor is configured to: obtain input associated with selecting a first object as a target object to be included in the adjusted second image.

[0140] Aspect 8: The apparatus of any of Aspects 1 to 7, wherein the input associated with selecting the first object as the target object to be included in the adjusted second image includes at least one or more of physical contact, gesture, gaze direction, input from an input device, or audio input.

[0141] Aspect 9: An apparatus of any of Aspects 1 to 8, wherein the processor is configured to: determine that the second object is an unwanted object, wherein determining that the second object is an unwanted object includes at least one or more of the following: determining that the second object moves between the first image of the scene and the second image of the scene; determining that the first object is a previously photographed object and the second object is not a previously photographed object; or determining that the depth of the first object and the depth of the second object differ by an amount greater than a threshold depth.

[0142] Aspect 10: The apparatus of any of Aspects 1 to 9, wherein removing the second object from the adjusted second image comprises interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

[0143] Aspect 11: The apparatus of any of Aspects 1 to 10, wherein removing the second object from the adjusted second image comprises acquiring pixels of the first image of the scene associated with the third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0144] Aspect 12: The apparatus of any one of Aspects 1 to 11, wherein the first image of the scene and the second image of the scene are preview images acquired before acquiring the capture input.

[0145] Aspect 13: The apparatus of any one of Aspects 1 to 12, wherein the second image of the scene is an image acquired after acquiring the capture input.

[0146] Aspect 14: An apparatus of any of Aspects 1 to 13, wherein the processor is configured to: obtain a capture input; obtain a third image of the scene, wherein the third image of the scene includes the first object at the first position and the second object; determine a fourth position of the second object in the scene; generate an adjusted third image including the first object at the first position, wherein the second object is removed from the adjusted third image; and store the third image.

[0147] Aspect 15: An apparatus of any of Aspects 1 to 14, wherein the processor is configured to: obtain a segmentation associated with a second object and a first image of the scene; associate a third image of the scene with the first image of the scene; and apply the segmentation associated with the second object and the first image of the scene to the third image of the scene.

[0148] Aspect 16: The apparatus of any one of Aspects 1 to 15, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution.

[0149] Aspect 17: An apparatus of any of Aspects 1 to 16, wherein, to generate an adjusted second image, the at least one processor is configured to: perform inline processing to remove the second object from the second image of the scene; and generating an adjusted third image includes combining pixels of a fourth image of the scene associated with a fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene occluded by the second object in the third image of the scene.

[0150] Aspect 18: The apparatus of any one of Aspects 1 to 17, wherein the third image and the fourth image are stored in a ZSL buffer.

[0151] Aspect 19: The apparatus of any of Aspects 1 to 18, wherein the processor is configured to: acquire a fourth image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position, wherein the fourth image of the scene includes an image captured independently of acquiring a capture input; generate an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object located at the first position and the second object is removed from the adjusted fourth image; and display the adjusted fourth image on a display.

[0152] Aspect 20: The apparatus of any one of Aspects 1 to 19, wherein the processor is configured to: obtain input associated with removing the second object based on the first image of the scene.

[0153] Aspect 21: The apparatus of any one of Aspects 1 to 20, wherein the second object is removed based on an inline process.

[0154] Aspect 22: An apparatus of any of Aspects 1 to 21, wherein the processor is configured to: acquire a fifth image of a scene from a camera; acquire a capture input; and based on acquiring the capture input, generate a captured image based on the fifth image of the scene, wherein the captured image includes a first object located at a first position and a second object located at a second position is removed from the captured image; and store the captured image in at least one memory.

[0155] Aspect 23: A method for processing one or more images, comprising: acquiring a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position; acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object located at the first position and the second object located at a third position, the third position being different from the second position; generating an adjusted second image based on the second image, wherein the adjusted second image includes the first object located at the first position and the second object at the third position is removed from the adjusted second image; and displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image include images captured independently of acquiring a capture input.

[0156] Aspect 24: The method of Aspect 23 further includes: acquiring a third image of the scene from the camera; acquiring a capture input; and generating a captured image based on the third image of the scene based on the acquired capture input, wherein the captured image includes a first object located at the first position, and a second object located at the third position is removed from the captured image.

[0157] Aspect 25: The method of any one of Aspects 23 to 24, wherein the second image of the scene and the third image of the scene are associated with a common identifier.

[0158] Aspect 26: The method of any of Aspects 23 to 25, wherein the third image of the scene comprises an image acquired before the input is captured, wherein the third image of the scene is stored in memory.

[0159] Aspect 27: The method of any one of Aspects 23 to 26, wherein the memory comprises a zero shutter delay (ZSL) buffer.

[0160] Aspect 28: The method of any one of Aspects 23 to 27, further comprising, after displaying the first image of the scene on the display, obtaining input associated with selecting a second object to be removed from the scene.

[0161] Aspect 29: The method of any one of Aspects 23 to 28, further comprising obtaining input associated with selecting a first object as a target object to be included in the adjusted second image.

[0162] Aspect 30: The method of any of Aspects 23 to 29, wherein the input associated with selecting the first object as the target object to be included in the adjusted second image includes at least one or more of physical contact, gesture, gaze direction, input from an input device, or audio input.

[0163] Aspect 31: The method of any of Aspects 23 to 30 further includes determining that the second object is an unwanted object, wherein determining that the second object is an unwanted object includes at least one or more of the following: determining that the second object moves between the first image of the scene and the second image of the scene; determining that the first object is a previously photographed object and the second object is not a previously photographed object; or determining that the depth of the first object and the depth of the second object differ by an amount greater than a threshold depth.

[0164] Aspect 32: The method of any of Aspects 23 to 31, wherein removing the second object from the adjusted second image comprises interpolating pixel colors of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

[0165] Aspect 33: The method of any of Aspects 23 to 32, wherein removing the second object from the adjusted second image comprises acquiring pixels of the first image of the scene associated with the third location and replacing corresponding pixels of the second image of the scene associated with the third location.

[0166] Aspect 34: The method of any one of Aspects 23 to 33, wherein the first image of the scene and the second image of the scene are preview images acquired before acquiring the capture input.

[0167] Aspect 35: The method of any one of Aspects 23 to 34, wherein the second image of the scene is an image acquired after acquiring the capture input.

[0168] Aspect 36: The method of any of Aspects 23 to 35, further comprising: obtaining a capture input; obtaining a third image of the scene, wherein the third image of the scene includes the first object at the first position and the second object; determining a fourth position of the second object in the scene; generating an adjusted third image including the first object at the first position, wherein the second object is removed from the adjusted third image; and storing the third image.

[0169] Aspect 37: The method of any of Aspects 23 to 36, further comprising: obtaining a segmentation associated with the second object and the first image of the scene; associating a third image of the scene with the first image of the scene; and applying the segmentation associated with the second object and the first image of the scene to the third image of the scene.

[0170] Aspect 38: The method of any one of Aspects 23 to 37, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution.

[0171] Aspect 39: The method of any of Aspects 23 to 38, wherein generating an adjusted second image includes performing an inline process to remove a second object from the second image of the scene; and generating an adjusted third image includes combining pixels of a fourth image of the scene associated with a fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene occluded by the second object in the third image of the scene.

[0172] Aspect 40: The method of any one of Aspects 23 to 39, wherein the third image and the fourth image are stored in a ZSL buffer.

[0173] Aspect 41: The method of any of Aspects 23 to 40, further comprising: acquiring a fourth image of the scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position, wherein the fourth image of the scene includes an image captured independently of acquiring a capture input; generating an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object located at the first position and the second object is removed from the adjusted fourth image; and displaying the adjusted fourth image on a display.

[0174] Aspect 42: The method of any one of Aspects 23 to 41, further comprising obtaining input associated with removing the second object based on the first image of the scene.

[0175] Aspect 43: The method of any one of Aspects 23 to 42, wherein the second object is removed based on an inline process.

[0176] Aspect 44: The method of any of Aspects 23 to 43 further includes: acquiring a fifth image of the scene from a camera; acquiring a capture input; and generating a captured image based on the fifth image of the scene based on acquiring the capture input, wherein the captured image includes a first object located at a first position and a second object located at a second position is removed from the captured image; and storing the captured image.

[0177] Aspect 45: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations of Aspects 1 to 44.

[0178] Aspect 46: An apparatus comprising one or more means for performing any of the operations of Aspects 1 to 44.

Claims

1. An apparatus for processing one or more images, comprising: at least one memory; as well as at least one processor coupled to the at least one memory and configured to: acquiring a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position; acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object at the first position and the second object at a third position, the third position is different from the second position; generating an adjusted second image based on the second image, wherein the adjusted second image includes the first object at the first location and the second object at the third location is removed from the adjusted second image; as well as The adjusted second image is displayed on a display, wherein the first image of the scene and the adjusted second image comprise images captured independently of obtaining a capture input.

2. The apparatus of claim 1 , wherein the at least one processor is configured to: acquiring a third image of the scene from the camera; obtaining the captured input; and Based on obtaining the capture input, a captured image based on the third image of the scene is generated, wherein the captured image includes the first object located at the first position and the second object located at the third position is removed from the captured image.

3. The apparatus of claim 2, the second image of the scene and the third image of the scene being associated with a common identifier. 4 . The apparatus of claim 3 , the third image of the scene comprising an image acquired prior to the capture input, wherein the third image of the scene is stored in the at least one memory.

5. The apparatus of claim 4, the at least one memory comprising a zero shutter delay (ZSL) buffer.

6. The apparatus of claim 1, wherein the at least one processor is configured to obtain input associated with selecting to remove the second object from the scene. 7 . The apparatus of claim 1 , wherein the at least one processor is configured to obtain input associated with selecting the first object to be included in the adjusted second image as a target object.

8. The apparatus of claim 7, the input associated with selecting the first object to be included as the target object in the adjusted second image comprising at least one or more of physical contact, gesture, gaze direction, input from an input device, or audio input.

9. The device according to claim 7, wherein: In order to determine that the second object is an unwanted object, the at least one processor is configured to: Determining that the second object is an unwanted object, wherein determining that the second object is an unwanted object comprises at least one or more of the following: determining that the second object moves between the first image of the scene and the second image of the scene; determining that the first object is a previously photographed object and the second object is not a previously photographed object; or It is determined that the depth of the first object differs from the depth of the second object by a difference greater than a threshold depth.

10. The apparatus of claim 1, wherein removing the second object from the adjusted second image comprises interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

11. The apparatus of claim 1 , removing the second object from the adjusted second image comprises acquiring pixels of the first image of the scene associated with the third position and replacing corresponding pixels of the second image of the scene associated with the third position. 12 . The apparatus of claim 1 , wherein the first image of the scene and the second image of the scene are preview images acquired before acquiring the capture input.

13. The apparatus of claim 1, the second image of the scene being an image acquired after acquiring the capture input.

14. The apparatus of claim 1, wherein the at least one processor is configured to: Get capture input; acquiring a third image of the scene, wherein the third image of the scene includes the first object and the second object at the first position; determining a fourth position of the second object in the scene; generating an adjusted third image including the first object at the first location, wherein the second object is removed from the adjusted third image; as well as The third image is stored.

15. The apparatus of claim 14, wherein the at least one processor is configured to: obtaining a segmentation associated with the second object and the first image of the scene; associating the third image of the scene with the first image of the scene; and The segmentation associated with the second object and the first image of the scene is applied to the third image of the scene.

16. The apparatus of claim 15, the first image having a first resolution and the third image having a second resolution, the second resolution being different from the first resolution.

17. The device of claim 14, wherein: To generate the adjusted second image, the at least one processor is configured to: performing an inpaint process to remove the second object from the second image of the scene; as well as Generating the adjusted third image comprises combining pixels of a fourth image of the scene associated with the fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene occluded by the second object in the third image of the scene.

18. The apparatus of claim 1, wherein the at least one processor is configured to: acquiring, from the camera, a fourth image of the scene, wherein the scene includes the first object located at the first location and the second object located at the second location, wherein the fourth image of the scene comprises an image captured independently of acquiring the capture input; generating an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object at the first location and the second object is removed from the adjusted fourth image; and The adjusted fourth image is displayed on a display.

19. The apparatus of claim 18, wherein the at least one processor is configured to obtain input associated with removing the second object based on the first image of the scene.

20. The apparatus of claim 18, wherein at least one processor is configured to: acquiring a fifth image of the scene from the camera; obtaining the captured input; and generating, based on obtaining a capture input, a captured image based on the fifth image of the scene, wherein the captured image includes the first object at the first location and the second object at the second location is removed from the captured image; and The captured image is stored in at least one memory.

21. A method of processing one or more images, comprising: acquiring a first image of a scene from a camera, wherein the scene includes a first object located at a first position and a second object located at a second position; acquiring a second image of the scene from the camera, wherein the second image of the scene includes the first object at the first position and the second object at a third position, the third position being different from the second position; generating an adjusted second image based on the second image, wherein the adjusted second image includes the first object at the first location and the second object at the third location is removed from the adjusted second image; as well as The adjusted second image is displayed on a display, wherein the first image of the scene and the adjusted second image comprise images captured independently of obtaining a capture input.

22. The method of claim 21, further comprising: acquiring a third image of the scene from the camera; Obtaining the captured input; as well as Based on obtaining the capture input, a captured image based on the third image of the scene is generated, wherein the captured image includes the first object located at the first position and the second object located at the third position is removed from the captured image.

23. The method of claim 22, wherein the second image of the scene and the third image of the scene are associated with a common identifier.

24. The method of claim 23, wherein the third image of the scene comprises an image acquired prior to the capture input, wherein the third image of the scene is stored in a memory.

25. The method of claim 24, wherein the memory comprises a zero shutter delay (ZSL) buffer.

26. The method of claim 21 further comprising, after displaying the first image of the scene on the display, obtaining input associated with selecting to remove the second object from the scene.

27. The method of claim 21, further comprising obtaining input associated with selecting the first object to be included in the adjusted second image as a target object.

28. The method of claim 27, wherein the input associated with selecting the first object to be included in the adjusted second image as the target object comprises at least one or more of physical contact, gesture, gaze direction, input from an input device, or audio input.

29. The method of claim 27, further comprising determining that the second object is an unwanted object, wherein determining that the second object is an unwanted object comprises at least one or more of the following: determining that the second object moves between the first image of the scene and the second image of the scene; determining that the first object is a previously photographed object and the second object is not a previously photographed object; or It is determined that the depth of the first object differs from the depth of the second object by a difference greater than a threshold depth.

30. The method of claim 21, wherein removing the second object from the adjusted second image comprises interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene.

31. The method of claim 21, wherein removing the second object from the adjusted second image comprises acquiring pixels of the first image of the scene associated with the third location and replacing corresponding pixels of the second image of the scene associated with the third location.

32. The method of claim 21, wherein the first image of the scene and the second image of the scene are preview images acquired prior to acquiring the capture input.

33. The method of claim 21, wherein the second image of the scene is an image acquired after acquiring the capture input.

34. The method of claim 21, further comprising: Get capture input; acquiring a third image of the scene, wherein the third image of the scene includes the first object and the second object at the first position; determining a fourth position of the second object in the scene; generating an adjusted third image including the first object at the first location, wherein the second object is removed from the adjusted third image; as well as The adjusted third image is stored.

35. The method of claim 34, further comprising: obtaining a segmentation associated with the second object and the first image of the scene; associating the third image of the scene with the first image of the scene; as well as The segmentation associated with the second object and the first image of the scene is applied to the third image of the scene.

36. The method of claim 35, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution.

37. The method of claim 34, wherein generating the adjusted second image comprises performing an inpainting process to remove the second object from the second image of the scene; and Generating the adjusted third image includes combining pixels of the fourth image of the scene associated with the fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene occluded by the second object in the third image of the scene.

38. The method of claim 21, further comprising: acquiring, from the camera, a fourth image of the scene, wherein the scene includes the first object located at the first location and the second object located at the second location, wherein the fourth image of the scene comprises an image captured independently of acquiring the capture input; generating an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object at the first location and the second object is removed from the adjusted fourth image; as well as The adjusted fourth image is displayed on the display.

39. The method of claim 38, further comprising obtaining input associated with removing the second object based on the first image of the scene.

40. The method of claim 38, further comprising: acquiring a fifth image of the scene from the camera; Obtaining the captured input; as well as generating, based on obtaining the capture input, a captured image based on the fifth image of the scene, wherein the captured image includes the first object at the first location and the second object at the second location is removed from the captured image; as well as The captured image is stored.

Images