Bystander and shadow removal

Abstract

The media application derives a bystander mask from the image by analyzing the image with a bystander segmentation model, the image depicting a bystander, and the bystander mask identifying a plurality of first pixels in the image associated with the bystander. The media derives a shadow mask for the bystander by analyzing the image with a shadow segmentation model, the image and the bystander mask being provided as inputs to the shadow segmentation model, and the shadow mask identifying a plurality of second pixels in the image associated with a shadow of the bystander. The media application modifies the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are removed from the image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 18 / 195,401, entitled "BYSTANDER AND ATTACHED SHADOW REMOVAL," filed May 10, 2023, the entire contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] The appeal of visual media items, such as images (e.g., still images, images with selective motion, etc.) and videos, can be improved by removing bystanders that distract from the focus of the media item. However, when a bystander is obscured by a shadow, pixels associated with the bystander can be identified and modified to remove the bystander, but pixels associated with the shadow remain. Previous techniques have attempted to solve this problem by generating a shadow mask, but accurately generating a shadow mask is difficult. For example, the process of identifying a shadow may result in false positives in the candidate shadow mask, where areas larger than the shadow are removed. Furthermore, in situations where a person in an image does not cast a shadow, the process of identifying a shadow may result in identifying a shadow that belongs to another object. When object classification is used for image editing purposes (e.g., replacing a bystander with pixels that match the background), the resulting image may be unsatisfactory due to the presence of pixels associated with the shadow that remain in the image even after the bystander is removed.

[0003] The description of the background art provided herein is intended to provide a general context for the present disclosure. To the extent provided in this background art section, the work of the inventors named herein, and aspects of the present disclosure that may not qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure. Summary of the Invention

[0004] The computer-implemented method includes deriving a bystander mask from an image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander. The method further includes deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, the image and the bystander mask being provided as inputs to the shadow segmentation model, the shadow mask identifying a plurality of second pixels in the image associated with a shadow of the bystander. The method further includes modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are removed from the image.

[0005] In some embodiments, the method further includes determining a likelihood that a shadow exists using a shadow classifier model based on the image and the bystander mask, and deriving a shadow mask for the bystander occurs if the likelihood that a shadow exists meets a threshold. In some embodiments, modifying the image includes applying an inpainting technique to update pixel values ​​of the plurality of first pixels and the plurality of second pixels. In some embodiments, the inpainting technique is performed by an inpainter model. In some embodiments, the method further includes merging the bystander mask and the shadow mask before modifying. In some embodiments, the shadow segmentation model includes a convolutional neural network trained using supervised learning, wherein training the convolutional neural network is done with a training dataset including a plurality of training images, a segmentation mask associated with a person in each training image of the plurality of training images, and a ground truth shadow mask for each training image of the plurality of training images, and the training includes: for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image and the segmentation mask associated with the person in the training image; calculating a loss value based on a comparison of the predicted shadow mask and the ground truth shadow mask for the image; and updating weights of one or more nodes of the convolutional neural network based on the loss value.In some embodiments, the training dataset is generated by including candidate images in a plurality of training images selected from a group of people associated with body bounding boxes that overlap less than a threshold value with the vehicle bounding box, body bounding boxes associated with an aspect ratio that is less than a threshold aspect ratio, a segmentation mask that is outside the vehicle bounding box, a body bounding box that intersects with the moving object bounding box less than a threshold overlap value, a ground truth shadow mask that is between a first size threshold and a second size threshold, a plurality of training images that satisfy an illumination threshold, candidate images that include shadows emanating from a direction associated with the person's feet, and combinations thereof. In some embodiments, one or more training images of the plurality of training images are associated with an empty ground truth shadow mask. In some embodiments, one or more training images of the plurality of training images are generated by: generating a clean plate image including static elements of the scene from a series of candidate images of a scene including people captured at different times; comparing each candidate image in the series of candidate images with the clean plate image to identify dynamic portions of the scene; generating a segmentation mask for the people in the series of images; and determining from the segmentation mask that pixels in a ground truth shadow mask are adjacent to pixels corresponding to the segmentation mask.In some embodiments, the shadow segmentation model includes a convolutional neural network trained using supervised learning, where training the convolutional neural network is performed using a training dataset including a plurality of training images and a ground truth shadow mask for each training image of the plurality of training images, where the training includes, for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image, where one or more pixels of the training image associated with a person have a value associated with a particular color that does not occur in natural images, and the training further includes calculating a loss value based on a comparison of the predicted shadow mask for the image with the ground truth shadow mask and updating weights of one or more nodes of the convolutional neural network based on the loss value. In some embodiments, the image depicts two or more bystanders, and the deriving, obtaining, and correcting are performed for each of the two or more bystanders. In some embodiments, the image is a single frame of a video. In some embodiments, at least one pixel of the plurality of first pixels of the bystander mask and at least one pixel of the plurality of second pixels of the shadow mask are adjacent.

[0006] In some embodiments, a non-transitory computer-readable medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including deriving a bystander mask from an image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander, the operations further including deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, the image and the bystander mask being provided as inputs to the shadow segmentation model, the shadow mask identifying a plurality of second pixels in the image associated with a shadow of the bystander, and the operations further including modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are removed from the image.

[0007] In some embodiments, the operations further include determining a likelihood that a shadow exists using a shadow classifier model based on the image and the bystander mask, and deriving a shadow mask for the bystander occurs if the likelihood that a shadow exists meets a threshold. In some embodiments, modifying the image includes applying an inpainting technique to update pixel values ​​of the plurality of first pixels and the plurality of second pixels. In some embodiments, the operations further include merging the bystander mask and the shadow mask before modifying.

[0008] In some embodiments, a computing device includes one or more processors and a memory coupled to the one or more processors, the memory storing instructions that, when executed by the processor, cause the processor to perform operations. The operations may include deriving a bystander mask from the image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander, the operations may further include deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, the image and the bystander mask being provided as inputs to the shadow segmentation model, the shadow mask identifying a plurality of second pixels in the image associated with a shadow of the bystander, and the operations may further include modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are erased from the image.

[0009] In some embodiments, the operations further include determining a likelihood that a shadow exists using a shadow classifier model based on the image and a bystander mask, and deriving a shadow mask for the bystander occurs if the likelihood that a shadow exists meets a threshold. In some embodiments, modifying the image includes applying an inpainting technique to update pixel values ​​of the plurality of first pixels and the plurality of second pixels.

[0010] The techniques described herein advantageously solve the problem of identifying bystander shadows for removal from images. The shadow segmentation model receives a person's mask as input and predicts the person's shadow mask. The bystander segmentation model is updated to use state-of-the-art person detection and segmentation instead of trying to make the shadow segmentation model detect people as accurately as state-of-the-art person detectors. The techniques further include methods for generating training datasets for training machine learning models that reduce false positives and include examples where bystanders do not cast shadows. Finally, the techniques include a segmentation pipeline with different classifiers to predict bystander shadows. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an example network environment for identifying shadows attached to bystanders, according to some embodiments described herein. [Figure 2] FIG. 1 is a block diagram of an example computing device for identifying a shadow cast on a bystander, according to some embodiments described herein. [Figure 3] 1 illustrates an example image, an example image with bystanders removed and no shadows removed, and an example image with both bystanders and shadows removed, according to some embodiments described herein. [Figure 4] FIG. 10 illustrates example images that are excluded from the training data, according to certain embodiments described herein. [Figure 5] 10A-10C illustrate example images in which candidate shadow masks are filtered out based on being too small or too large in size, according to certain embodiments described herein. [Figure 6] 1 illustrates an exemplary filtered image in which clouds are preventing a shadow from being cast, and an exemplary image with a shadow, according to some embodiments described herein. [Figure 7]1 shows an example image in which a person depicted in the image casts a shadow, according to some embodiments. [Figure 8] 1 illustrates an example candidate image with a ground truth shadow mask as part of the training data, according to some embodiments described herein. [Figure 9] 1 illustrates an example flowchart of a method for training a machine learning model to output a shadow mask, according to some embodiments described herein. [Figure 10A] 1 illustrates an example flowchart of a method for outputting a merged bystander mask and a shadow mask, according to some embodiments described herein. [Figure 10B] 10 shows an example flowchart of another method for outputting a merged bystander mask and a shadow mask, according to some embodiments described herein. [Figure 11] 1 illustrates an exemplary flowchart of a method for modifying an image to remove bystanders and their shadows from the image based on a derived bystander mask and a derived shadow mask, according to some embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] Exemplary Environment 100

[0013] FIG. 1 shows a block diagram of an exemplary environment 100 for identifying bystander shadows. The environment 100 includes a media server 101, a user device 115a, and a user device 115n coupled to a network 105. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 may include other servers or devices not shown in FIG. 1. In FIG. 1 and the remaining figures, a letter following a reference number, e.g., "115a," indicates a reference to the element with that particular reference number. A reference number in the text without a following letter, e.g., "115," indicates a general reference to an embodiment of the element bearing that reference number.

[0014] The media server 101 includes a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to a network 105 via signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, or fiber optic cable, or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technology. In some embodiments, the media server 101 transmits and receives data to and from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0015] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data associated with the user 125, the user preferences of the user 125, etc.

[0016] The user device 115 is a computing device that includes a memory coupled to a hardware processor. For example, the user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or any other electronic device that can access the network 105.

[0017] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored as media application 103b on user device 115a and / or as media application 103c on user device 115n. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, or fiber optic cable, or wireless connections, such as Wi-Fi, Bluetooth, or other wireless technologies. User devices 115a and 115n are accessed by users 125a and 125n, respectively. The user devices 115a and 115n in FIG. 1 are used as an example. While FIG. 1 shows two user devices 115a and 115n, the present disclosure applies to system architectures having one or more user devices 115.

[0018] The media application 103 is stored on the media server 101 or the user device 115. In some embodiments, operations described herein occur on the media server 101 or the user device 115. In some embodiments, some operations may occur on the media server 101 and some may occur on the user device 115. Performance of operations is subject to user settings. For example, the user 125a may specify settings that operations are to be performed on their respective device 115a rather than on the server 101. With such settings, operations described herein (e.g., with reference to FIGS. 9-11) occur entirely on the user device 115a, and no operations occur on the media server 101. Furthermore, the user 125a may specify that user images and / or other data be stored locally only on the user device 115a, and not on the media server 101. With such settings, user data is not transmitted to or stored on the media server 101. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of actions on such data by the media server 101 will occur only if the user consents to the transmission, storage, and performance of actions by the media server 101. The user will be provided with the option to change settings at any time, for example, so that the user can enable or disable use of the media server 101.

[0019] Machine learning models (e.g., neural networks or other types of models) are stored and used locally on the user device 115 with specific user permission when utilized for one or more operations. Server-side models are used only if authorized by the user. Additionally, trained models may be provided for use on the user device 115. During use, on-device training of the model may occur if authorized by the user 115. If authorized by the user 115, updated model parameters may be sent to the media server 101, for example, to enable federated learning. The model parameters do not include any user data.

[0020] In some embodiments, the media application 103 may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using software or a combination of hardware and software.

[0021] The media application 103 receives an image depicting a bystander. For example, the media application 103 receives the image from a camera that is part of the user device 115, or the media application 103 receives the image over the network 105. The media application 103 derives a bystander mask from the image by analyzing the image with a bystander segmentation model. The bystander mask identifies a plurality of first pixels in the image that are associated with the bystander.

[0022] In some embodiments, the image and the bystander mask are provided as inputs to a shadow segmentation model. In some embodiments, a shadow classifier model is applied to the image to determine whether the likelihood of a shadow being present in the image meets a threshold. In these embodiments, if the likelihood of a shadow being present in the image meets the threshold, the shadow segmentation model is used; otherwise, bystander removal occurs without using the shadow segmentation model. The shadow segmentation model derives a shadow mask for the bystander by analyzing the image. The shadow mask identifies a plurality of second pixels in the image that are associated with the bystander's shadow.

[0023] The media application 103 modifies the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that bystanders and shadows are removed from the image. In some embodiments, the media application 103 includes an inpainter model that applies an inpainting technique to update the pixel values.

[0024] Exemplary Computing Device 200

[0025] 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. Computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is a media server 101 used to run media application 103a. In another example, computing device 200 is a user device 115.

[0026] In some embodiments, computing device 200 includes a processor 235, memory 237, input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0027] Processor 235 may be one or more processors and / or processing circuits that execute program code and control basic operations of computing device 200. A “processor” includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), a system with multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing functionality, a dedicated processor for performing processing based on neural network models, neural circuitry, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors that implement neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce a probabilistic output; for example, the output produced by processor 235 may be inaccurate or accurate within a range from an expected output. Processing need not be limited to a particular geographic location or have time limitations. For example, a processor may perform its functions in real time, offline, in batch mode, etc. Portions of processing may be performed at different locations at different times by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0028] Memory 237 is typically provided in computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor or set of processors, and located separately from and / or integrated with processor 235. Memory 237 may store software operated on computing device 200 by processor 235, including media application 103.

[0029] Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in several environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0030] Application data 266 may be data generated by other applications 264 or by hardware of computing device 200. For example, application data 266 may include images used by an image library application and user actions identified by other applications 264 (e.g., social networking applications), etc.

[0031] I / O interface 239 may provide functionality that allows computing device 200 to interface with other systems and devices. The interfaced devices may be included as part of computing device 200 or may be separate and in communication with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate through I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0032] Some examples of interface devices that can be connected to I / O interface 239 can include display 241, which can be used to display content, such as images, videos, and / or user interfaces of output applications described herein, as well as to receive touch (or gesture) input from a user. For example, display 241 may be utilized to display a user interface including an image of bystanders and their shadows removed. Display 241 can include any suitable display device, such as a liquid crystal display (LCD), light-emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touchscreen, a three-dimensional display screen, or other visual display device. For example, display 241 can be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen for a computing device.

[0033] Camera 243 may be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 243 captures images or video that I / O interface 239 sends to media application 103.

[0034] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set including training data such as a plurality of training images, a bystander segmentation model, a shadow segmentation model, an inpainter model, etc.

[0035] 2 illustrates an exemplary media application 103, including a bounding box module 202, a bystander segmenter 204, a shadow segmenter 206, an inpainter module 208, and a user interface module 210. In some embodiments, each of the components includes a set of instructions executable by a processor 235 to perform steps described in more detail below. In some embodiments, each of the components may be stored in memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0036] Components of the media application 103 identify a bystander in the image, identify a shadow associated with the bystander, and update pixel values ​​associated with the bystander and shadow from the image to remove the bystander and shadow. FIG. 3 shows three example images. In a first example image 300, a bystander 305 and a shadow 310 associated with the bystander are shown (the primary subject of the image is a woman facing the camera that captured the image 300). In a second example image 325, the bystander has been removed, but the shadow 330 remains. This example is confusing and unnatural because the shadow 330 is not connected to the object, resulting in the unsatisfactory removal of the bystander 305. In a third example image 350, both the bystander and shadow have been removed, resulting in a satisfactory image. While the following description describes a single bystander and a single shadow, the description also applies to two or more bystanders associated with two or more corresponding shadows.

[0037] Turning to the bounding box module 202, the bounding box module 202 receives an image. The image may be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. In some embodiments, the image is a portion of a video, for example, a frame of a video.

[0038] The image includes a subject, such as a person, and one or more bystanders. The bounding box module 202 detects bystanders in the image. A bystander is a person who is not the subject of the image, such as a person walking, running, bicycling, or standing behind the subject or otherwise in the image. In different examples, a bystander may be in the foreground (e.g., a person visible in front of the subject, e.g., a person crossing between the camera 243 and the subject), at the same depth as the subject (e.g., a person standing next to the subject), or in the background (behind the subject). A bystander may be a human in any pose, e.g., standing, sitting, crouching, lying down, jumping, etc. A bystander may be facing the camera 243, at an angle to the camera 243, or facing away from the camera 243.

[0039] In some embodiments, the bounding box module 202 generates body bounding boxes for any people and / or bystanders in the image. For example, Figure 3 includes a person who is the subject of the image and a bystander. In other examples, the training images described in more detail below may include people but not bystanders.

[0040] In some embodiments, the body bounding box is a rectangular bounding box that encompasses all pixels of the person and / or bystander. The bounding box module 202 may detect the person and / or bystander by performing object recognition, comparing the object to objects in front of the person, and discarding non-person objects. In some embodiments, the bounding box module 202 uses machine learning algorithms, such as neural networks, or more specifically, convolutional neural networks, to identify the person and / or bystander and generate the body bounding box. The body bounding box is associated with the x and y coordinates of the bounding box.

[0041] The bounding box module 202 generates object bounding boxes that encompass objects that overlap with people and / or bystanders in the image. In some embodiments, the bounding box module 202 generates object bounding boxes for all objects in the image and then identifies object bounding boxes for objects that overlap with bystanders in the image. Intersecting object bounding boxes include, for example, people riding bicycles, people riding scooters, people in vehicles, etc.

[0042] The bystander segmenter 204 analyzes the image with the bystander segmentation model to derive a bystander mask from the image. A bystander mask encompassing the bystander is generated. The bystander segmenter 204 identifies a plurality of first pixels in the image associated with the bystander. In some embodiments, the bystander segmenter 204 identifies the plurality of first pixels in the image based on analyzing pixels within a body bounding box generated by the bounding box module 202.

[0043] In some embodiments, the bystander mask is generated based on generating superpixels of the image and matching the centroids of the superpixels to depth map values ​​(e.g., obtained by the camera 243 using a depth sensor or by deriving depth from pixel values) for depth-based cluster detection. More specifically, the depth values ​​within the masked region may be used to determine a depth range, and superpixels that fall within the depth range may be identified. However, the depth of objects attached to the bystander may be the same as the ground, resulting in an over-inclusive bystander mask.

[0044] Other techniques for generating a bystander mask include weighting depth values ​​based on how close the depth values ​​would be to the bystander mask if the weights were represented by a distance transform map. However, the bystander mask may over-cover the ground and under-cover parts of the object.

[0045] The shadow segmenter 206 receives the image and the bystander mask as input to a shadow segmentation model that analyzes the image and derives a shadow mask for the bystander, where the shadow mask identifies a number of second pixels in the image that are associated with the bystander's shadow.

[0046] In some embodiments, the shadow segmentation model is a trained machine learning model. In some embodiments, the shadow segmenter 206 is configured to apply the bystander segmentation model to input data, such as application data 266 (e.g., images captured by the user device 115), to output a bystander mask.

[0047] The shadow segmenter 206 generates a shadow segmentation model using training data. Model generation may be performed offline (e.g., on the media server 101 or other server), with the shadow segmentation model provided as part of the shadow segmenter 206. For example, the training data may include training images with objects (e.g., people, buildings, gardens, etc.) and shadows cast on the objects (positive examples), as well as training images with people not casting shadows (negative examples). In some embodiments, the training data further includes body bounding boxes surrounding bystanders and object bounding boxes for objects in the images.

[0048] The training images may be RGB (red-green-blue) images in which the color of each pixel is characterized by three components R, G, and B. In some embodiments, one or more of the RGB images are associated with a segmentation mask corresponding to people in the RGB images. In some embodiments, one or more of the RGB images are associated with a body bounding box surrounding the people. In some embodiments, one or more of the RGB images include pixels of people that have values ​​associated with particular colors that do not occur in natural images. For example, the shadow segmenter 206 may use a flood-fill algorithm on the segmentation mask to identify and / or modify neighboring values ​​based on their similarity to an initial seed point associated with an uncommon color. In some embodiments, the flood-fill algorithm is used to locate people in the image using uncommon colors instead of perturbing the person pixels in the RGB image.

[0049] The training data may be obtained from any source, e.g., a data repository specifically marked for training, data that has been given permission to be used as training data for machine learning, etc. In some embodiments, training may occur on the media server 101 providing the training data directly to the user device 115, training may occur locally on the user device 115, or a combination of both.

[0050] In some embodiments, the training data may be obtained from a series of images (e.g., a time lapse or image burst) of a scene captured at different times. For example, the series of images may be obtained from security camera footage. The shadow segmenter 206 may extract 3D camera position information, organize the images based on the same 3D camera position, and apply a median filter (or mode filter) to the pixels over time to generate a clean plate image based on the median or mode of pixel values. The median filter removes dynamic objects, such as moving vehicles, people, and animals, from the image. As a result, the clean plate image represents static elements of the scene, such as buildings, sidewalks, and roads.

[0051] The shadow segmenter 206 compares each frame in the series of images with the clean plate image to identify dynamic portions of the scene that contain people and their shadows. The shadow segmenter 206 then uses the identification of dynamic portions of the scene to identify potential people in each image by performing object recognition. The shadow segmenter 206 generates a segmentation mask for the person. The shadow segmenter 206 uses a connected component algorithm (e.g., breadth-first search) on the differences between the clean plate image and each frame to identify all dynamic pixels that are adjacent to pixels corresponding to the segmentation mask. These dynamic pixels form a high-quality mask of the person's shadow, called a ground truth shadow mask, that is part of the training data.

[0052] In some embodiments, the shadow segmenter 206 uses criteria to determine whether a candidate training image is included or excluded from the training images, for example, the shadow segmenter 206 identifies candidate images to include as part of the training images based on the intersection between a body bounding box and an object bounding box.

[0053] The shadow segmenter 206 may select a candidate image for inclusion in the training images if the bystander is associated with a body bounding box that has an overlap value below a threshold with the vehicle bounding box or a ground truth bystander mask that is inside the vehicle bounding box. For example, Figure 4 shows a first exemplary image 400 and a second exemplary image 410, where the body bounding boxes 402, 412 in images 400, 410 are inside the vehicle's object boxes 404, 424, and therefore images 400, 410 are excluded from the training images. In this example, the threshold overlap value may be 100%, although other values ​​are possible, such as a value indicating the bystander is too close to the vehicle, partially inside the vehicle, etc.

[0054] The shadow segmenter 206 may select a candidate image to include as part of the training images if the bystander is associated with a body bounding box that is less than a threshold aspect ratio. The shadow segmenter 206 determines the aspect ratio by dividing the width of the body bounding box by the height of the body bounding box. If the aspect ratio exceeds the threshold aspect ratio, the candidate image is likely to be occluded by an object in the candidate image.

[0055] The shadow segmenter 206 may select a candidate image to include as part of the training images if the bystander is associated with a body bounding box that intersects with a moving object bounding box by less than a threshold overlap value. The moving object may be a bicycle, scooter, tricycle, unicycle, skateboard, etc. Figure 4 shows a third example image 450 and a fourth example image 460 in which the body bounding box 452, 462 intersects with the bicycle bounding box 454, 464 by more than the threshold overlap value, and therefore the images 450, 460 are excluded from the training images.

[0056] The shadow segmenter 206 may select candidate images for inclusion as training images whose ground truth shadow masks are between a first size threshold and a second size threshold. The first size threshold may be for ground truth shadow masks that are too small, and the second size threshold may be for ground truth shadow masks that are too large. For example, FIG. 5 shows a first image 500 and a second image 525, both of which include ground truth shadow masks 505, 530 that are excluded from the training images for not exceeding the first size threshold because their shadow masks are too small. FIG. 5 also shows a third image 550 and a fourth image 575, both of which include ground truth shadow masks 555, 580. Because the shadow masks 555 and 580 are larger than the second threshold, images 550 and 575 are excluded from the training images for exceeding the second size threshold.

[0057] The shadow segmenter 206 may select a candidate image for inclusion as a training image if the candidate image meets the illumination threshold. Using the illumination threshold ensures that images taken under certain conditions, such as during a cloudy day, are not used because these conditions prevent shadow formation. FIG. 6 shows a first image 600 in which clouds result in no shadows being cast by objects in the scene; the first image 600 does not meet the illumination threshold. As a result, the shadow segmenter 206 excludes the first image 600 from the training images. Conversely, the second image 625 in FIG. 6 contains sufficient illumination to meet the illumination threshold, and the tree-related shadow 630 is very noticeable.

[0058] The shadow segmenter 206 may select a candidate image for inclusion as a training image if the candidate image contains a shadow emanating from a direction associated with the feet of a person in the image. In some embodiments, the shadow segmenter 206 determines the direction the shadow should emanate from based on the geographic location on Earth and date and time corresponding to the candidate image's metadata. For example, at a particular date, time, and location, the sun casts a shadow at a particular angle, and the shadow segmenter 206 may determine whether the shadow is associated with the person or something else in the image, such as a nearby object.

[0059] The shadow segmenter 206 may select a candidate image for inclusion as a training image if the candidate image does not contain a shadow. A person may not cast a shadow for several reasons, including the person standing in a larger shadow, such as a shadow cast by a building, the shadow not being visible because, for example, the person's feet are blocked, or the lighting of the scene is overcast and not in direct sunlight. Figure 7 shows example images without shadows, where a first image 700 contains a person walking in a larger shadow 705 covering part of a street, a second image 725 contains a scene where the person does not cast a shadow due to overcast conditions, and a third image 750 contains a vehicle 755 blocking the person's feet.

[0060] In some embodiments, the shadow segmenter 206 may select candidate images that do not contain shadows for inclusion as training images by running a shadow extraction algorithm to determine whether the calculated shadow does not contain a threshold number of pixels (e.g., at least one pixel, at least five pixels, or other threshold). In some embodiments, the shadow segmenter 206 determines that a candidate image does not contain a shadow by determining whether the scene was captured during cloudy weather conditions, such as by applying an illumination model. In some embodiments, the shadow segmenter 206 determines that a candidate image does not contain a shadow by marking the image with a label indicating that the image does not contain a shadow, or by asking human raters to identify applicable images.

[0061] In some embodiments, the shadow segmenter 206 selects candidate images for inclusion as training images based on human feedback. For example, a human may rate the images on a scale (e.g., 1-4, good vs. bad, etc.). In some embodiments, a human rater receives the candidate images after the shadow segmenter 206 selects them for inclusion as training images using the criteria described above. Referring to FIG. 8, exemplary candidate images 800, 825, 850 are shown, with ground truth shadow masks 805, 830, 855 included as part of the training data.

[0062] The trained machine learning model trained by shadow segmenter 206 may include one or more model formats or structures. For example, the model format or structure may include any type of neural network, such as a linear network, a deep learning neural network implementing multiple layers (e.g., with a "hidden layer" between the input and output layers, where each layer is a linear network), a convolutional neural network (e.g., a network that divides or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of processing each tile), a sequence-to-sequence neural network (e.g., a network that receives sequence data as input, such as words in a sentence or frames in a video, and produces a resulting sequence as output), etc.

[0063] The model format or structure may specify the connectivity between various nodes and their organization into layers. For example, nodes in a first layer (e.g., input layer) may receive data as input data or application data 266. Such data may include, for example, one or more pixels per node, for example, when the trained model is used to analyze images. Subsequent intermediate layers may receive as input the outputs of nodes in the previous layer, per the connections specified in the model format or structure. These layers are sometimes referred to as hidden layers. The final layer (e.g., output layer) produces the output of the machine learning model. In some implementations, the model format or structure also specifies the number and / or type of nodes in each layer.

[0064] In different implementations, the trained shadow segmentation model may include one or more models. One or more of the models may include multiple nodes arranged in layers according to a model structure or format. In some implementations, a node may be, for example, a memoryless computational node configured to process a unit of input and produce a unit of output. The computation performed by the node may include, for example, multiplying each of multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce the node output. In some implementations, the computation performed by the node may also include applying a step function / activation function to the adjusted weighted sum. In some implementations, the step function / activation function may be a nonlinear function. In various implementations, such computations may include operations such as matrix multiplication. In some implementations, computations by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, individual processing units of a graphics processing unit (GPU), or dedicated neural circuitry. In some implementations, a node may include memory, for example, capable of storing and using one or more previous inputs when processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node, which may use memory to maintain a "state" that allows the node to function like a finite state machine (FSM).

[0065] In some implementations, the trained model may include embeddings or weights for individual nodes. For example, the model may begin as a plurality of nodes organized into layers, as specified by the model format or structure. At initialization, respective weights may be applied to nodes connected according to the model format, e.g., to the connection between each pair of nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The model may then be trained, e.g., using training data, to produce results.

[0066] The training may include applying a supervised learning method. In supervised learning, the training data may include multiple inputs (e.g., multiple training images) and corresponding ground truth outputs for each input (e.g., a ground truth shadow mask for each training image of the multiple training images). Based on a comparison of the model's output (e.g., predicted shadow mask) and the ground truth output (e.g., ground truth shadow mask), the values ​​of the weights are automatically adjusted, for example, in a manner that increases the probability that the model will produce the ground truth shadow output for the image.

[0067] In various implementations, the trained model includes a set of weights or embeddings corresponding to the model structure. In some implementations, the trained shadow segmentation model may include an initial set of weights, for example, downloaded from a server that provides the weights. In various implementations, the trained shadow segmentation model includes a set of weights or embeddings corresponding to the model structure. In implementations where data is omitted, the shadow segmenter 206 may generate a trained shadow segmentation model based on prior training, for example, by the developer of the shadow segmenter 206, by a third party, etc.

[0068] In some embodiments, if the shadow segmentation model includes a convolutional neural network trained using supervised learning, training the shadow segmentation model may include, for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image. The shadow segmentation model may calculate a loss value based on a comparison of the predicted shadow mask for the image with a ground truth shadow mask (included in the training data). The shadow segmentation model may update weights of one or more nodes of the convolutional neural network based on the loss value (e.g., so that the loss value decreases after performing another cycle of adjustment and training until the loss value is below a threshold).

[0069] FIG. 9 shows an example flowchart of a method 900 for training a machine learning model to output a shadow mask. An RGB image of a person and a segmentation mask 905 are provided as input to a machine learning model 910, such as the shadow segmentation model described above. In some embodiments, the machine learning model 910 is a convolutional neural network (CNN) such as UNet. The machine learning model 910 outputs a predicted shadow mask 915. The predicted shadow mask 915 is compared to a ground truth shadow mask 920 to determine a loss value 925. In some embodiments, the loss value 925 is calculated using binary cross-entropy loss, which is a comparison of each predicted probability with the actual closest output, which is either 0 or 1. The loss value 925 is calculated as a score reflecting the distance between the predicted probability and the actual value. The score is used to update the weights of the model during training. The process may continue with additional training images until the score meets a threshold score (or the training data / computation budget is exhausted) and training is complete.

[0070] In some embodiments, when the shadow segmentation model includes a convolutional neural network trained using supervised learning, training the shadow segmentation model may include, for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image in which one or more pixels of the training image associated with a person have values ​​associated with a particular color that do not occur in natural images. The shadow segmentation model may calculate a loss value based on a comparison of the predicted shadow mask and a ground truth shadow mask for the image. The shadow segmentation model may update weights of one or more nodes of the convolutional neural network based on the loss value.

[0071] In some embodiments, the shadow segmenter 206 receives the image and a bystander mask. The shadow segmenter 206 provides the image and the bystander mask as inputs to a shadow segmentation model. In some embodiments, the shadow segmentation model outputs a shadow mask.

[0072] 10A shows an exemplary flowchart of a method 1000 for outputting a merged bystander mask and shadow mask. In step 1005, bystander detection is performed. For example, in this case, a girl 1010 is detected to the right of a person 1015, the subject of the image. For each bystander, a person segmenter 1020 is applied and a shadow segmenter 1025 is applied. The person segmenter 1020 outputs a bystander mask 1022, and the shadow segmenter 1025 outputs a shadow mask 1027. In step 1030, the shadow mask and the bystander mask are merged.

[0073] 10B shows an exemplary flowchart of another method 1050 for outputting a merged bystander mask and shadow mask. In step 1055, bystander detection is performed. For example, in this case, a girl 1060 is detected to the right of a person 1065, the subject of the image. For each bystander, a person segmenter 1070 is applied, a shadow classifier 1075 determines whether the bystander is casting a shadow, and if so, a shadow segmenter 1080 is applied. The person segmenter 1070 outputs a bystander mask 1072, the shadow classifier 1075 determines that the bystander is casting a shadow, and the shadow segmenter 1080 outputs a shadow mask 1082. In step 1085, the shadow mask and the bystander mask are merged. If the shadow classifier 1075 determines that the bystander is not casting a shadow, the bystander mask is used as is for further tasks.

[0074] In some embodiments, the inpainter module 208 modifies the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that bystanders and shadows are removed from the image. The inpainter module 208 may apply an inpainting technique to update pixel values ​​of the plurality of first pixels and the plurality of second pixels. In some embodiments, the inpainter module 208 applies the inpainting technique using an inpainter model.

[0075] In some embodiments, the inpainter module 208 generates an inpainted image that updates pixel values ​​of pixels in the bystander mask and the shadow mask to match the background in the image such that bystanders and shadows are erased from the image. The inpainter module 208 may update pixel values ​​of a plurality of first pixels that are part of the bystander mask separately from pixel values ​​of a plurality of second pixels that are part of the shadow mask. In some embodiments, the inpainter module 208 merges the bystander mask and the shadow mask to create a combined mask, and the inpainter module 208 updates pixel values ​​of pixels in the combined mask such that bystanders and shadows are erased from the image.

[0076] Pixels matching the background may be based on other images of the same location without the subject and / or bystander. Alternatively, the inpainter module 208 may match pixels removed from the bystander mask based on pixels surrounding the pixels included in the bystander mask. For example, if a bystander was standing on the ground, the inpainter module 208 replaces the pixels with ground pixels, such as in the third image 350 of FIG. 3. Other inpainting techniques are possible, including machine learning-based inpainter techniques.

[0077] In some embodiments, the techniques described above may be applied to video. For example, a component of the media application 103 may derive bystander and shadow masks for a first image and then apply temporal smoothing between images to maintain consistency in the mask pixels or slowly change the masks across multiple frames so that there are no discontinuities across frames. In some embodiments, optical flow associated with a person is used to determine the direction of motion, and the mask pixels are updated accordingly across frames of the video. In some embodiments, bystander and shadow masks are determined for a subset of frames, and interpolation is used to extend the masks across the entire video.

[0078] The user interface module 210 generates the user interface. In some embodiments, the user interface module 210 includes a set of instructions executable by the processor 235 for generating the user interface. In some embodiments, the user interface module 210 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0079] The user interface module 210 generates a user interface that includes the inpainted image. In some embodiments, the user interface includes options for editing the inpainted image, sharing the inpainted image, adding the inpainted image to a photo album, etc.

[0080] Exemplary Flowchart

[0081] 11 shows an example flowchart of a method 1100 of modifying an image to remove bystanders and their shadows from the image based on a derived bystander mask and a derived shadow mask. Method 1100 may be performed by computing device 200 of FIG. 2. In some embodiments, method 1100 is performed by user device 115, media server 101, or partially on user device 115 and partially on media server 101 of FIG. 1.

[0082] 11 may begin at block 1102. In block 1102, a bystander mask is derived from an image by analyzing the image with a bystander segmentation model. The image depicts a bystander, and the bystander mask identifies a plurality of first pixels in the image associated with the bystander. Block 1102 may be followed by block 1104.

[0083] In block 1104, an optional step is to determine the likelihood that a shadow exists using a shadow classifier model based on the image and the bystander mask. If the likelihood that a shadow exists meets a threshold, block 1104 may be followed by block 1106.

[0084] At block 1106, a shadow mask for the bystander is derived by analyzing the image with a shadow segmentation model. The image is provided as an input to the shadow segmentation model. The shadow mask identifies a plurality of second pixels in the image that are associated with the bystander's shadow.

[0085] In some embodiments, the shadow segmentation model includes a convolutional neural network trained using supervised learning, wherein training the convolutional neural network is performed using a training dataset including a plurality of training images and a ground truth shadow mask for each training image of the plurality of training images. The training may include, for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image and a ground truth shadow mask associated with a person in the training image, calculating a loss value based on a comparison of the predicted shadow mask for the image and the ground truth shadow mask, and updating weights of one or more nodes of the convolutional neural network based on the loss value. In some embodiments, the training includes, for each training image of the plurality of training images, obtaining a predicted shadow mask based on the training image, wherein one or more pixels of the training image associated with the person have a value associated with a particular color that does not occur in natural images, and the training further includes calculating a loss value based on a comparison of the predicted shadow mask for the image with the ground truth shadow mask, and updating weights of one or more nodes of the convolutional neural network based on the loss value.

[0086] In some embodiments, the plurality of training images further includes a segmentation mask corresponding to a person in each training image of the plurality of training images, and the training dataset is generated by including candidate images in the plurality of training images selected from a group of people associated with body bounding boxes that overlap less than a threshold value with the vehicle bounding box, body bounding boxes associated with aspect ratios less than a threshold aspect ratio, segmentation masks that are outside the vehicle bounding box, body bounding boxes that intersect with the moving object bounding box less than a threshold overlap value, a ground truth shadow mask that is between a first size threshold and a second size threshold, a plurality of training images that satisfy an illumination threshold, candidate images that include shadows emanating from a direction associated with the person's feet, and combinations thereof. In some embodiments, one or more training images of the plurality of training images are associated with a sky ground truth shadow mask. In some embodiments, the sky ground truth shadow mask identifies zero pixels as being part of a shadow cast by a person, thereby ensuring that the training images include negative examples with no shadows (e.g., images taken at noon, images in which a person is in the shadow of a larger object such as a building).

[0087] In some embodiments, one or more training images of the plurality of training images are generated by: generating a clean plate image including static elements of the scene from a series of candidate images of a scene including people captured at different times; comparing each candidate image in the series of candidate images with the clean plate image to identify dynamic portions of the scene; generating a segmentation mask for the people in the series of images; and determining from the segmentation mask that pixels in a ground truth shadow mask are adjacent to pixels corresponding to the segmentation mask.

[0088] In some embodiments, the image depicts two or more bystanders, and the derived, acquired, and modified images are performed for each of the two or more bystanders. In some embodiments, the image is a single frame of video. Block 1106 may be followed by block 1108.

[0089] In block 1108, an optional step is merging the bystander mask and the shadow mask. Block 1108 may be followed by block 1110. At least one pixel of the plurality of first pixels of the bystander mask and at least one pixel of the plurality of second pixels of the shadow mask may be adjacent.

[0090] In block 1110, the image is modified to update pixel values ​​of a plurality of first pixels and a plurality of second pixels (e.g., using an inpainting technique) so that bystanders and shadows are erased from the image. Block 1110 may be followed by block 1112.

[0091] In block 1112, an optional step is to apply an inpainting technique to update pixel values ​​of the plurality of first pixels and the plurality of second pixels. The inpainting technique may be performed by an inpainter model.

[0092] In addition to the above, users may be provided with controls that allow them to make choices about both whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location), as well as whether the user is sent content or communications from the server. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be determined. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0093] In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form in order to avoid obscuring the specification. For example, embodiments may be described above primarily with reference to user interfaces and specific hardware. However, embodiments may apply to any type of computing device capable of receiving data and commands, and any peripheral device that provides services.

[0094] A reference herein to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with an embodiment or instance may be included in at least one implementation herein. The appearances of the phrase "in some embodiments" in various places in this specification are not necessarily all referring to the same embodiments.

[0095] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0096] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, as will be apparent from the discussion that follows, throughout this specification, discussions utilizing terms including "processing," "calculating," "figuring out," "determining," or "displaying," etc., will be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that is also represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0097]

[0013] Embodiments herein may also relate to a processor for performing one or more steps of the above-described methods. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory computer-readable storage medium, including, but not limited to, any type of disk, including an optical disk; a ROM, a CD-ROM, a magnetic disk, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each connected to a computer system bus.

[0098] The specification may take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments containing both hardware and software elements, hi some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0099] Furthermore, the specification may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, instruction execution apparatus, or instruction execution device.

[0100] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory utilized during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during execution.

Claims

1. 1. A computer-implemented method comprising: and deriving a bystander mask from the image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander, the computer-implemented method further comprising: deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, wherein the image and the bystander mask are provided as inputs to the shadow segmentation model, and the shadow mask identifies a plurality of second pixels in the image associated with a shadow of the bystander; modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are eliminated from the image.

2. 2. The computer-implemented method of claim 1, further comprising determining a likelihood that the shadow exists using a shadow classifier model based on the image and the bystander mask, wherein deriving the shadow mask for the bystander occurs if the likelihood that the shadow exists meets a threshold.

3. 2. The computer-implemented method of claim 1, wherein modifying the image comprises applying an inpainting technique to update the pixel values ​​of the plurality of first pixels and the plurality of second pixels.

4. The computer-implemented method of claim 3 , wherein the inpainting technique is performed by an inpainter model.

5. The computer-implemented method of claim 1 , further comprising merging the bystander mask and the shadow mask before said modifying.

6. The shadow segmentation model includes a convolutional neural network trained using supervised learning, wherein training the convolutional neural network is performed with a training dataset including a plurality of training images, a segmentation mask associated with a person in each training image of the plurality of training images, and a ground truth shadow mask for each training image of the plurality of training images, and wherein the training includes, for each training image of the plurality of training images: obtaining a predicted shadow mask based on the training images and the segmentation mask associated with the person in the training images; calculating a loss value based on a comparison of the predicted shadow mask and the ground truth shadow mask for the image; updating weights of one or more nodes of the convolutional neural network based on the loss value; 2. The computer-implemented method of claim 1, comprising:

7. 7. The computer-implemented method of claim 6, wherein the training dataset is generated by including candidate images in the plurality of training images selected from the group of the person associated with a body bounding box that has less than a threshold overlap value with a vehicle bounding box, the body bounding box associated with an aspect ratio that is less than a threshold aspect ratio, the segmentation mask that is outside a vehicle bounding box, the body bounding box that intersects with a moving object bounding box that has less than the threshold overlap value, the ground truth shadow mask that is between a first size threshold and a second size threshold, the plurality of training images that satisfy an illumination threshold, the candidate images that include shadows emanating from a direction associated with the person's feet, and combinations thereof.

8. The computer-implemented method of claim 6 , wherein one or more training images of the plurality of training images are associated with an empty ground truth shadow mask.

9. One or more of the training images of the plurality of training images are selected from a series of candidate images of scenes including people captured at different times. generating a clean plate image containing static elements of the scene; comparing each candidate image in the series of candidate images with the clean plate image to identify dynamic portions of the scene; generating the segmentation mask for the person in the sequence of images; and determining from the segmentation mask that pixels of the ground truth shadow mask are adjacent to pixels corresponding to the segmentation mask.

10. The shadow segmentation model includes a convolutional neural network trained using supervised learning, wherein training the convolutional neural network is performed with a training dataset including a plurality of training images and a ground truth shadow mask for each training image of the plurality of training images, and the training includes, for each training image of the plurality of training images: obtaining a predicted shadow mask based on the training images, wherein one or more pixels of the training images associated with a person have values ​​associated with a particular color that does not occur in natural images, and the training further comprises, for each training image of the plurality of training images: calculating a loss value based on a comparison of the predicted shadow mask and the ground truth shadow mask for the image; updating weights of one or more nodes of the convolutional neural network based on the loss value; 2. The computer-implemented method of claim 1, comprising:

11. 2. The computer-implemented method of claim 1, wherein the images depict two or more bystanders, and the deriving, obtaining, and modifying are performed for each of the two or more bystanders.

12. The computer-implemented method of claim 1 , wherein the image is a single frame of video.

13. 2. The computer-implemented method of claim 1, wherein at least one pixel of the plurality of first pixels of the bystander mask and at least one pixel of the plurality of second pixels of the shadow mask are adjacent.

14. A non-transitory computer-readable medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including: deriving a bystander mask from the image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander, the operations further comprising: deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, wherein the image and the bystander mask are provided as inputs to the shadow segmentation model, and the shadow mask identifies a plurality of second pixels in the image associated with a shadow of the bystander, the operations further comprising: modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are eliminated from the image.

15. 15. The non-transitory computer-readable medium of claim 14, wherein the operations further include determining a likelihood that the shadow is present using a shadow classifier model based on the image and the bystander mask, and deriving the shadow mask for the bystander is performed if the likelihood that the shadow is present meets a threshold.

16. 15. The non-transitory computer-readable medium of claim 14, wherein modifying the image comprises applying an inpainting technique to update the pixel values ​​of the plurality of first pixels and the plurality of second pixels.

17. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise merging the bystander mask and the shadow mask before the modifying.

18. 1. A computing device comprising: a processor; a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: deriving a bystander mask from the image by analyzing the image with a bystander segmentation model, the image depicting a bystander, the bystander mask identifying a plurality of first pixels in the image associated with the bystander, the operations further comprising: deriving a shadow mask for the bystander by analyzing the image with a shadow segmentation model, wherein the image and the bystander mask are provided as inputs to the shadow segmentation model, and the shadow mask identifies a plurality of second pixels in the image associated with a shadow of the bystander, the operations further comprising: modifying the image to update pixel values ​​of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are eliminated from the image.

19. 20. The computing device of claim 18, wherein the operations further include determining a likelihood that the shadow is present using a shadow classifier model based on the image and the bystander mask, and deriving the shadow mask for the bystander is performed if the likelihood that the shadow is present meets a threshold.

20. 20. The computing device of claim 18, wherein modifying the image comprises applying an inpainting technique to update the pixel values ​​of the plurality of first pixels and the plurality of second pixels.

Images