Removal of bystanders and accessory objects

Abstract

The media application detects a bystander in the initial image. The media application generates a bystander box that includes the bystander, where all pixels of the bystander are within the bystander box. The media application generates a localizer box that encompasses the bystander and one or more objects attached to the bystander. The media application aggregates the bystander box and one or more of the localizer boxes to form an aggregate box. The media application applies a segmenter to the initial image based on the aggregate box to segment the bystander and the one or more objects from the initial image to generate a bystander mask, where the bystander mask includes a subset of the pixels within the aggregate box. The media application generates an inpaint image that replaces all pixels in the bystander mask with pixels that match a background in the initial image.

Description

[Background technology]

[0001] background Classification of objects in visual media items such as images (still images, images with selective motion, etc.) and videos can be improved by removing bystanders that distract from the focus of the media item. However, when bystanders are attached to an object (e.g., a person riding a bicycle / skateboard, a person carrying a bag, etc.), pixels associated with the bystander may be identified and pixels associated with the object may not be identified. When such classification is used for image editing purposes (e.g., replacing bystanders with pixels that match the background), the resulting image may be unsatisfactory due to the presence of pixels associated with the object.

[0002] The background description provided herein is intended to generally present the context of the present disclosure. The inventors' work to the extent described in this Background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not expressly or impliedly admitted as prior art to the present disclosure. Summary of the Invention [Means for solving the problem]

[0003] overview The computer-implemented method comprises detecting a bystander in an initial image. The method further comprises generating a bystander box that includes the bystander, where all pixels of the bystander are within the bystander box. The method further comprises generating a localizer box that encompasses the bystander and one or more objects attached to the bystander. The method further comprises aggregating one or more of the bystander box and the localizer box to form an aggregate box. The method further comprises applying a segmenter to the initial image based on the aggregate box to segment the bystander and one or more objects from the initial image to generate a bystander mask, where the bystander mask includes a subset of pixels within the aggregate box. The method further comprises generating an inpaint image, where all pixels in the bystander mask are replaced in the inpaint image with pixels that match the background in the initial image. In one example, the method or at least one or more steps of the method may be performed or provided by a machine learning model. The machine learning model may be, for example, a trained machine learning model obtained by a training method described below. Thus, the machine learning model may perform at least one of the steps of generating a localizer box, forming an aggregation box, applying a segmenter, and optionally generating an inpaint image. The machine learning model may further perform any of the method steps described below. Instead of using just one model, multiple models can also be used.

[0004] In some embodiments, the method further comprises: prior to applying the segmenter to the initial image, defining the aggregate box to be a bystander box; for each of the localizer boxes, calculating an intersection over union (IoU) ratio of the aggregate box to the localizer box; and increasing a size of the aggregate box in response to the IoU ratio exceeding a threshold or in response to an overlap area of ​​the aggregate box and the localizer box exceeding an occupancy threshold. In some embodiments, the IoU ratio is the overlap area of ​​the aggregate box and the localizer box divided by the union area of ​​the aggregate box and the localizer box, and the method further comprises padding the aggregate box. In some embodiments, applying the segmenter to the initial image includes segmenting a foreground from a background to distinguish one or more objects pertaining to the bystander from the background. In some embodiments, the method further comprises determining whether the object in the initial image occludes a bystander, and in response to determining that the object in the initial image occludes a bystander, generating an object mask and updating the bystander mask to remove pixels in the object mask. In some embodiments, the segmenter is a trained machine learning model. In some embodiments, the localizer box includes one or more of the object, bystander, or object in the initial image, and the method further comprises removing a corresponding localizer box associated with the object.

[0005] In some embodiments, a computer-implemented method (e.g., also referred to as a training method) comprises obtaining a synthetic training dataset generated to include training images each including an object overlaid on a background image, and obtaining a trained machine learning model, where obtaining the trained machine learning model is performed by providing the synthetic training dataset as input to a raw machine learning model, outputting predicted bystander boxes for the training images, comparing each predicted bystander box to a ground truth bystander mask corresponding to the training images, and adjusting one or more parameters of the raw machine learning model based on the comparison, where the trained machine learning model is trained to receive an initial image and output a bystander mask including a subset of pixels within the aggregate box.

[0006] In some embodiments, the trained machine learning model is a neural network including multiple layers and outputs a bystander mask in the trained machine learning model, the bystander mask including a first layer that segments the image into foreground and background by classifying the pixel as either part of the foreground or part of the background for each pixel, and an output layer that outputs for each pixel whether the pixel is part of the bystander mask, part of the subject mask, or neither, the first layer preceding the output layer in the neural network. In some embodiments, the method further includes receiving an initial image having an aggregate box that encompasses the bystander and one or more objects attached to the bystander as an input to the trained machine learning model, outputting a bystander mask including a subset of pixels in the aggregate box using the trained machine learning model, and generating an inpaint image, where all pixels in the bystander mask are replaced in the inpaint image with pixels that match the background in the initial image. In some embodiments, the method further comprises detecting a bystander in the initial image prior to receiving the initial image as an input to the machine learning model, and generating a bystander box including the bystander, where all pixels of the bystander are within the bystander box, and the method further comprises generating a localizer box encompassing the bystander and one or more objects attached to the bystander, and aggregating one or more of the bystander box and the localizer box to form an aggregate box. In some embodiments, the method further comprises defining the aggregate box to be a bystander box, calculating, for each of the bystander boxes, an intersection over union (IoU) ratio of the aggregate box to the localizer box, and increasing a size of the aggregate box in response to the IoU ratio exceeding a threshold or in response to an overlap area between the aggregate box and the localizer box exceeding an occupancy threshold.In some embodiments, the IoU ratio is an overlap area of ​​the aggregate box and the localizer box divided by an area of ​​a union of the aggregate box and the localizer box, and the method further comprises padding the aggregate box. In some embodiments, outputting the bystander mask includes outputting a corresponding confidence value and adding blur around edges of the bystander in response to the corresponding confidence value not satisfying a confidence threshold.

[0007] In some embodiments, a computing device comprises one or more processors and a memory coupled to the one or more processors storing instructions that, when executed by the processor, cause the processor to perform operations. The operations may include detecting a bystander in an initial image and generating a bystander box that includes the bystander, where all pixels of the bystander are within the bystander box, and the operations may further include generating a localizer box that encompasses the bystander and one or more objects attached to the bystander, aggregating one or more of the bystander box and the localizer box to form an aggregate box, and applying a segmenter to the initial image based on the aggregate box to segment the bystander and the one or more objects from the initial image to generate a bystander mask, where the bystander mask includes a subset of the pixels within the aggregate box, and the operations further include generating an inpaint image, where all pixels in the bystander mask are replaced in the inpaint image with pixels that match a background in the initial image.

[0008] In some embodiments, the operations further include, prior to applying the segmenter to the initial image, defining the aggregate box to be a bystander box, and for each of the localizer boxes, calculating an intersection over union (IoU) ratio of the aggregate box to the localizer box, and increasing a size of the aggregate box in response to the IoU ratio exceeding a threshold or in response to an overlap area of ​​the aggregate box and the localizer box exceeding an occupancy threshold. In some embodiments, the IoU ratio is the overlap area of ​​the aggregate box and the localizer box divided by the area of ​​the union of the aggregate box and the localizer box, and the operations further include padding the aggregate box. In some embodiments, applying the segmenter to the initial image includes segmenting a foreground from a background to distinguish one or more objects pertaining to the bystander from the background. In some embodiments, the operations further include determining whether an object in the initial image occludes a bystander, and in response to determining that an object in the initial image occludes a bystander, generating an object mask and updating the bystander mask to remove pixels in the object mask, In some embodiments, the segmenter is a trained machine learning model.

[0009] The techniques described herein advantageously solve the problem of identifying objects to remove from an image when the object is attached to a bystander by generating an aggregation box that includes the bystander and one or more objects attached to the bystander, an bystander mask that includes a subset of the pixels within the aggregation box, and an inpaint image that replaces pixels in the bystander mask with pixels that match the background in the initial image. In some embodiments, when a subject in the initial image occludes a bystander, the present application describes techniques for generating a subject mask in which pixels that are part of the subject mask, generated based on a pixel mask that does not include subject pixels in the bystander mask, are retained in the final image. The described techniques can automatically perform image editing to remove the bystander and attached objects, or other actions such as blurring the bystander and attached objects, to generate the final image. Effect of the Invention

[0010] Thus, the method of the present disclosure solves various technical problems. For example, the method can provide bystander-removed images that can improve more reliable and / or accurate detection of the subject of the initial image (e.g., by using the same model or another model with the respective capabilities). In another example, the bystander-removed provided images can be used as an optimization data set for training another machine learning model (e.g., for person detection or image classification). This data set can be optimized because confusing bystanders that may confuse the model in the training task have been removed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example network environment according to some embodiments described herein. [Diagram 2] FIG. 1 is a block diagram illustrating an exemplary computing device according to some embodiments described herein. [Figure 3A] FIG. 2 illustrates an exemplary initial image according to some embodiments described herein. [Figure 3B] FIG. 2 illustrates an example initial image with bystander boxes and localizer boxes according to some embodiments described herein. [Figure 3C] FIG. 1 illustrates an example initial image with a box aggregator output, according to some embodiments described herein. [Figure 3D] FIG. 2 illustrates an example initial image with aggregate boxes and bystander boxes according to some embodiments described herein. [Figure 3E] FIG. 2 illustrates an exemplary initial image with a bystander mask that encompasses bystanders and attached objects according to some embodiments. [Figure 3F] FIG. 2 illustrates an exemplary final image with bystanders and objects removed according to some embodiments described herein. [Figure 4] 1 is an exemplary flowchart for generating an in-paint image with bystanders and attached objects removed according to some embodiments described herein. [Diagram 5] 1 is an example flowchart for generating a bystander mask according to some embodiments described herein. [Figure 6] FIG. 2 illustrates the operation of an example neural network, according to some embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Detailed Description Exemplary Environment 100 FIG. 1 is a block diagram illustrating an example environment 100. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Each of the users 125a, 125n may be associated with a user device 115a, 115n. In some embodiments, the environment 100 may include other servers or devices not shown in FIG. 1. In FIG. 1 and the remaining figures, a letter following a reference number, e.g., "115a," denotes a reference to the element having that particular reference number. In the text, a reference number without a letter following the number, e.g., "115," denotes a general reference to an embodiment of the element having that reference number.

[0013] The media server 101 may include a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to a network 105 via signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technology. In some embodiments, the media server 101 transmits and receives data to and from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0014] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data related to the users 125, user preferences of the users 125, etc.

[0015] The user device 115 may be a computing device that includes a memory coupled to a hardware processor. For example, the user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or other electronic device that can access the network 105.

[0016] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored as media application 103b on user device 115a and / or media application 103c on user device 115n. Signal lines 108 and 110 may be wired connections such as Ethernet, coaxial cable, fiber optic cable, or wireless connections such as Wi-Fi, Bluetooth, or other wireless technologies. User devices 115a, 115n are accessed by users 125a, 125n, respectively. User devices 115a, 115n in FIG. 1 are used as an example. Although FIG. 1 shows two user devices 115a, 115n, the present disclosure applies to a system architecture having one or more user devices 115.

[0017] The media application 103 may be stored on the media server 101 or on the user device 115. In some embodiments, the operations described herein are executed on the media server 101 or on the user device 115. In some embodiments, some operations may be executed on the media server 101 and some on the user device 115. The execution of the operations is subject to user settings. For example, the user 125a may specify settings to execute operations on the respective device 115a and not on the server 101. Such settings result in the operations described herein (e.g., with reference to Figures 4-6) being executed entirely on the user device 115a and no operations being executed on the media server 101. Additionally, the user 125a may specify that the user's images and / or other data be stored only locally on the user device 115a and not on the media server 101. Such settings result in no user data being transmitted to or stored on the media server 101. The transmission of user data to the media server 101, the temporary or permanent storage of that data by the media server 101, and the performance of operations on that data by the media server 101 occur only if the user consents to the transmission, storage, and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example to enable or disable the use of the media server 101.

[0018] Machine learning models (e.g., neural networks or other types of models) are stored locally on the user device 115 and utilized with the permission of a particular user when utilized for one or more operations. Server-side models are used only if authorized by the user. Model training is performed using a synthetic data set, as described below with reference to FIG. 5. Additionally, trained models may be provided for use on the user device 115. During such use, on-device training of the model may be performed if authorized by the user 125. Updated model parameters may be transmitted to the media server 101 if authorized by the user 115, for example, to enable federated learning. Model parameters do not include user data.

[0019] The media application 103 receives an image. For example, the media application 103 receives an image from a camera that is part of the user device 115, or the media application 103 receives an image over the network 105. The media application 103 detects one or more bystanders in the initial image and generates an bystander box that includes the bystander, where all pixels of the bystander are within the bystander box. The media application 103 further generates one or more localizer boxes that encompass one or more objects attached to the bystander, and aggregates the bystander box and the one or more localizer boxes to form an aggregate box.

[0020] The media application 103 applies a segmenter to the initial image based on the aggregation box to segment the bystander and the one or more objects from the initial image to generate a bystander mask, the bystander mask including a subset of the pixels within the aggregation box, and generates an inpaint image that replaces all pixels in the bystander mask with pixels that match the background in the initial image.

[0021] In some embodiments, the media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / co-processor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using a combination of hardware and software.

[0022] Exemplary Computing Device 200 2 is a block diagram illustrating an example computing device 200 that may be used to implement one or more features described herein. The computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 used to implement a media application 103a. In another example, the computing device 200 is a user device 115.

[0023] In some embodiments, computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all coupled via a bus 218. Processor 235 may be coupled to bus 218 via signal line 222. Memory 237 may be coupled to bus 218 via signal line 224. I / O interface 239 may be coupled to bus 218 via signal line 226. Display 241 may be coupled to bus 218 via signal line 228. Camera 243 may be coupled to bus 218 via signal line 230. Storage device 245 may be coupled to bus 218 via signal line 232.

[0024] The processor 235 may be one or more processors and / or processing circuits that execute program code and control basic operations of the computing device 200. A "processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a system including a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing a function, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 235 may include one or more co-processors that implement neural network processing. In some embodiments, the processor 235 may be a processor that processes data to generate a probabilistic output, e.g., the output generated by the processor 235 may be inaccurate or may be accurate within a range from an expected output. The processing need not be limited to a particular geographic location, nor need it have a time restriction. For example, the processor may perform its functions in real time, offline, in batch mode, etc. Parts of the processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor in communication with a memory.

[0025] Memory 237 is typically provided in computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), Electrically Erasable Read-only Memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by a processor or set of processors, and may be located separately from and / or integrated with processor 235. Memory 237 may store software operated on computing device 200 by processor 235, including media application 103.

[0026] Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application ("app") operating on a mobile computing device, etc.

[0027] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions identified by other applications 264 (e.g., social networking applications), and the like.

[0028] The I / O interface 239 may provide functionality that allows the computing device 200 to interface with other systems and devices. The interfaced devices may be included as part of the computing device 200 or may be separate and in communication with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate through the I / O interface 239. In some embodiments, the I / O interface 239 may connect to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0029] Some examples of interfaced devices that can be connected to I / O interface 239 may include a display 241 that can be used to display content, e.g., images, video, and / or a user interface of an output application as described herein, and receive touch (or gesture) input from a user. For example, display 241 may be utilized to display a user interface including a graphical guide on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, display 241 may be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen of a computing device.

[0030] The camera 243 may be any type of image capture device capable of capturing images and / or video. In some embodiments, the camera 243 captures images or video that the I / O interface 239 sends to the media application 103.

[0031] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store training data sets including labeled images, machine learning models, output from machine learning models, and the like.

[0032] FIG. 2 illustrates an exemplary media application 103 that includes a bounding box module 202 , a segmenter 204 , a machine learning module 206 , an inpainter module 208 , and a user interface module 210 stored in memory 237 .

[0033] The bounding box module 202 generates a bounding box. In some embodiments, the bounding box module 202 includes a set of instructions executable by the processor 235 to generate a bounding box. In some embodiments, the bounding box module 202 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0034] In some embodiments, the bounding box module 202 receives an initial image. The initial image may be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. The initial image includes a subject, such as a person. The bounding box module 202 detects bystanders in the initial image. Bystanders are people who are not subjects of the initial image, such as people walking, running, bicycling, standing behind the subject, or people who are otherwise in the initial image. In different examples, bystanders may be in the foreground (e.g., a person passing in front of the camera), at the same depth as the subject (e.g., a person standing to the side of the subject), or in the background. In some examples, there may be multiple bystanders in the initial image. Bystanders may be people in any pose, such as standing, sitting, crouching, lying, jumping. Bystanders may be facing the camera, at an angle to the camera, or facing away from the camera. For example, Figure 3A shows an exemplary initial image 300 of a subject and a bystander. The subject is a woman walking and the bystander is a man riding a scooter.

[0035] The bounding box module 202 generates a bystander box that includes the bystander. In some embodiments, the bystander box is a rectangular bounding box that encompasses the pixels of the bystander. The bounding box module 202 may detect bystanders by performing object recognition, comparing objects to objects in front of the person, and discarding non-human objects. In some embodiments, the bounding box module 202 uses machine learning algorithms, such as neural networks, more specifically convolutional neural networks, to identify bystanders and generate bystander boxes. The bystander box is associated with the x and y coordinates of the bounding box.

[0036] Bystanders in the initial image are attached to objects. For example, bystanders may be riding a bicycle, standing next to or holding a bag, holding an umbrella, riding a scooter, etc. The bounding box module 202 generates bounding boxes for objects in the initial image. Objects may include non-human objects such as purses, strollers, scooters, and people and / or parts of people such as a human face and a human pants. Objects may also include pets, stationary objects (e.g., street lights, trash cans, etc.), or other types of objects. If a bounding box is associated with a location (e.g., x and y coordinates of the bounding box, x and y coordinates of the center of the bounding box, y shift and x shift indicating distance to the edge of the bounding box, etc.) and annotations of object type, the bounding box may be called a localizer box.

[0037] The bounding box module 202 generates a localizer box that encompasses the bystander and one or more objects associated with the bystander in the initial image. For example, FIG. 3B shows an example initial image 310 with a bystander box, as indicated by the dashed box around a man on a scooter, and localizer boxes. The various localizer boxes encompass objects such as a messenger bag worn by the bystander, clothing items such as the pants of the bystander and the pants of the subject 312, as well as the face of the bystander and the face of the subject 314 in the initial image.

[0038] The bounding box module 202 removes the localizer boxes associated with the objects of the initial image, which were the localizer boxes 312 and 314 in Figure 3B. Figure 3C shows an example initial image 320 with the localizer boxes associated with the objects of the initial image removed.

[0039] The bounding box module 202 aggregates the bystander box and the localizer box to form an aggregate box. The bounding box module 202 may determine whether the aggregate box is large enough to encompass the bystander and the object. In some embodiments, the bounding box module 202 compares the area of ​​the aggregate box to the area of ​​each of the localizer boxes. For example, with reference to FIG. 3C , the bounding box module 202 compares the area of ​​the dashed box around the bystander to a solid box 322 that encompasses the bystander, the messenger bag, and the scooter, compares the area of ​​the dashed box to a solid box 323 that encompasses the scooter, and compares the area of ​​the dashed box to a solid box 324 that encompasses the messenger bag.

[0040] The bounding box module 202 calculates an intersection over union (IoU) ratio, which is the ratio of the overlap area of ​​the aggregate box with each of the localizer boxes divided by the area of ​​the union of the aggregate box with each of the localizer boxes. For each of the localizer boxes, if the IoU ratio exceeds a threshold or if the overlap area of ​​the aggregate box with each of the localizer boxes exceeds an occupancy threshold, the bounding box module 202 increases the size of the aggregate box until the threshold or occupancy threshold is met.

[0041] In some embodiments, once the size of the aggregate box is finalized, the bounding box module 202 pads the aggregate box. For example, the bounding box module 202 pads the aggregate box with a predetermined amount of pixels (e.g., 10 pixels) outside the aggregate box to ensure that the aggregate box is inclusive. In some implementations, the padding may be equal on all four sides of the aggregate box. In some implementations, different amounts of padding may be used on different sides (e.g., 0 pixels on the left, 5 pixels above and below, 10 pixels on the right, etc.). FIG. 3D illustrates an example initial image 330 with an aggregate box 332 and a bystander box. The aggregate box 332 is larger than the dashed bystander box (which encompasses the bystander) because it encompasses the bystander and the attached objects, the messenger bag and the scooter.

[0042] In some embodiments, the media application 103 includes a segmenter 204 that generates a bystander mask. In some embodiments, the media application 103 includes a machine learning module 206 that generates a trained machine learning model that segments the initial image and outputs a bystander mask. In some embodiments, the segmenter 204 is used when there is no overlap between the bystander and the subject of the initial image, and the machine learning module 206 is used when there is overlap between the bystander and the subject. In some embodiments, the segmenter 204 also employs machine learning techniques or works in conjunction with the machine learning module 206.

[0043] The segmenter 204 segments bystanders and one or more objects within an aggregate box of the initial image to generate a bystander mask. In some embodiments, the segmenter 204 includes a set of instructions executable by the processor 235 to generate a bystander mask. In some embodiments, the segmenter 204 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0044] A bystander mask is generated that is inclusive of the bystander and one or more objects to be removed from the initial image. This process is more difficult when the bystander is attached to one or more objects, because it is difficult to identify the object attached to the bystander. In some embodiments, the bystander mask is generated based on generating superpixels on the input image and matching centroids of the superpixels to depth map values ​​to perform depth-based clustering detection. More specifically, the depth values ​​in the masked region may be used to determine a depth range and identify superpixels that fall within the depth range. However, since the depth of the object attached to the bystander is the same as the ground, this may result in an overly inclusive bystander mask.

[0045] Another technique for generating a bystander mask involves weighting depth values ​​based on how close the depth values ​​were to a bystander mask represented by a distance transform map, where the weights are a function of the distance transform map, but the bystander mask may over-include the ground and under-include parts of the object.

[0046] In some embodiments, segmenter 204 solves such technical problems by receiving an initial image that includes an aggregation box and segmenting the foreground from the background to distinguish objects attached to bystanders from the background, such as a sidewalk. Segmenter 204 uses pixels of bystanders and objects attached to bystanders within the aggregation box to generate a bystander mask. The subset of pixels used for the bystander mask is smaller than the pixels within the aggregation box.

[0047] In some embodiments, the task of generating a bystander mask is further complicated by the object occluding a portion of the bystander. For example, referring to FIG. 3A, the right arm of the object overlaps with the scooter attached to the bystander. If the segmenter 204 generates the bystander mask without considering the object, the bystander mask may include a portion of the object, and the imparter module may remove a portion of the subject's arm. In some embodiments, the segmenter 204 determines whether an object in the initial image occludes a bystander, and in response to the object in the initial image occluding a bystander, generates the object mask and updates the bystander mask to remove pixels in the object mask.

[0048] The machine learning module 206 generates a trained model, referred to herein as a machine learning model. In some embodiments, the machine learning module 206 is configured to apply the machine learning model to input data, such as application data 266 (e.g., an initial image captured by the user device 115), to output a bystander mask. In some embodiments, the machine learning module 206 may include software code executed by the processor 235. In some embodiments, the machine learning module 206 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0049] In some embodiments, the machine learning module 206 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 235 to apply the machine learning model. In some embodiments, the machine learning module 206 may include software instructions, hardware instructions, or a combination. In some embodiments, the machine learning module 206 may provide an application programming interface (API) that can be used by the operating system 262 and / or other applications 264 to invoke the machine learning module 206, e.g., to apply the machine learning model to application data 266 and output a segmentation mask.

[0050] The machine learning module 206 uses the training data to generate a trained machine learning model. For example, the training data may include images having a subject, a bystander, one or more objects attached to the bystander, and an aggregation box, in which the bystander and one or more objects are segmented and associated with a bystander mask. In some implementations, the aggregation box may be manually generated for each image in the training data.

[0051] The training data may come from any source, e.g., a data repository specially marked for training, data that has been given permission to be used as training data for machine learning, etc. In some embodiments, training may occur on the media server 101 that provides the training data directly to the user device 115, training may occur locally on the user device 115, or it may be a combination of both.

[0052] In some embodiments, the training data may include synthetic data generated for training purposes, such as data that is not based on activity in the context being trained, e.g., data generated from simulated or computer-generated images / videos. The training data may include synthetic images of subjects partially occluding bystanders. In some embodiments, the synthetic images are generated by overlaying a two-dimensional or three-dimensional object onto a background image. The three-dimensional object may be rendered from a particular view to convert the three-dimensional object into a two-dimensional object. In some embodiments, the synthetic images are generated by overlaying multiple people and objects onto a background to emulate compositing or occlusion effects. Different masks, such as people masks and object masks, may be added to the synthetic images to train the raw machine learning model to distinguish between people and objects in the synthetic images.

[0053] In some embodiments, the machine learning module 206 uses weights obtained and unedited / transferred from another application. For example, in these embodiments, the trained model may be generated, for example, on a different device and provided as part of the media application 103. In various embodiments, the trained model may be provided as a data file that includes a model structure or format (e.g., defining the number and type of neural network nodes, the connectivity between the nodes, and the organization of the nodes into multiple layers) and associated weights. The machine learning module 206 may read the trained model data file and implement a neural network with node connectivity, layers, and weights based on the model structure or format specified in the trained model.

[0054] The trained machine learning model may include one or more model formats or structures. For example, the model formats or structures may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or segments input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of processing each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence or frames in a video, and produces a result sequence as output), etc.

[0055] The model format or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, the nodes of the first layer (e.g., input layer) may receive data as input data or application data. Such data may include, for example, one or more pixels per node, if the trained model is used, for example, to analyze an initial image. Subsequent intermediate layers may receive the output of the nodes of the previous layer as input according to the connectivity specified in the model format or structure. These layers may also be referred to as hidden layers. For example, the first layer may output a segmentation between foreground and background. The final layer (e.g., output layer) generates the output of the machine learning model. For example, the output layer may receive the segmentation of the initial image into foreground and background and output whether the pixel is part of an observer mask, a subject mask, or neither. In some implementations, the model format or structure also specifies the number and / or type of nodes in each layer.

[0056] In different implementations, the trained model may include one or more models. One or more of the models may include multiple nodes arranged in layers per model structure or format. In some implementations, the node may be, for example, a memory-less computational node configured to process one unit of input and generate one unit of output. The computation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In some implementations, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a nonlinear function. In various implementations, such computations may include operations such as matrix multiplication. In some implementations, the computations by the multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or using special-purpose neural circuits. In some implementations, a node may include a memory, e.g., store and use one or more previous inputs when processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node. An LSTM node may use memory to maintain a "state" that allows the node to operate like a finite state machine (FSM).

[0057] In some implementations, the trained model may include embeddings or weights for individual nodes. For example, the model may begin as a number of nodes organized into layers as specified by the model format or structure. At initialization, respective weights may be applied to nodes that are connected according to the model format, e.g., to connections between each pair of nodes in successive layers of a neural network. For example, respective weights may be randomly assigned or initialized to default values. The model may then be trained to generate results, e.g., using training data.

[0058] Training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., images, aggregation boxes, person masks, etc.) and corresponding ground truth outputs for each input (e.g., ground truth bystander masks that correctly identify bystanders and attached objects for each image). Based on a comparison of the model's output to the ground truth outputs, the values ​​of the weights are automatically adjusted, for example, to increase the probability that the model generates the ground truth output for an image.

[0059] In various implementations, the trained model includes a set of weights, or embeddings, that correspond to the model structure. In some implementations, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights. In various implementations, the trained model includes a set of weights, or embeddings, that correspond to the model structure. In implementations where data is omitted, the machine learning module 206 may generate a trained model based on prior training, e.g., by a developer of the machine learning module 206, a third party, etc. In some implementations, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0060] In some embodiments, the machine learning module 206 receives an initial image having a subject, a bystander associated with one or more objects, and an aggregation box surrounding the bystander and one or more objects. The machine learning module 206 provides the initial image as an input to a trained machine learning model. In some embodiments, the trained machine learning model outputs a bystander mask that includes a subset of the pixels in the aggregation box. Figure 3E shows an exemplary initial image 340 with a bystander mask (illustrated by a solid black line) that encompasses the bystander and attached objects.

[0061] In some embodiments, the machine learning model outputs a confidence value for each bystander mask output by the trained machine learning model. The confidence value may be expressed as a percentage, a number between 0 and 1, etc. For example, the machine learning model outputs a confidence value of 85% for confidence that the bystander mask correctly incorporates bystanders and attached objects and does not contain pixels from the subject. In some embodiments, if the confidence value is below a threshold, the imparter module 208 applies a blurring effect or other effect around the edges of the bystander to minimize any errors.

[0062] In some embodiments, the machine learning module 206 receives feedback from users on the user devices 115. The feedback may take the form of users posting in-painted images, removing in-painted images, sharing in-painted images, etc. The machine learning module 206 modifies parameters of the machine learning model based on the feedback.

[0063] The inpainter module 208 generates an inpaint image that replaces all pixels in the bystander mask with pixels that match the background in the initial image. In some embodiments, the inpainter module 208 includes a set of instructions executable by the processor 235 to generate the inpaint image. In some embodiments, the inpainter module 208 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0064] In some embodiments, the inpainter module 208 receives an initial image with a bystander mask from the machine learning module 206. The inpainter module 208 replaces all pixels in the bystander mask with pixels that match the background in the initial image. The pixels that match the background may be based on another image of the same location without the subject and / or bystander. Alternatively, the inpainter module 208 may match pixels that are removed from the bystander mask based on pixels surrounding the bystander mask. For example, if the bystander was standing on the ground, the inpainter module 208 replaces the pixels with ground pixels. FIG. 3F illustrates an example inpaint image 350 with the bystander and object removed. Other inpainting techniques are possible, including machine learning based inpainter techniques.

[0065] The user interface module 210 generates a user interface. In some embodiments, the user interface module 210 includes a set of instructions executable by the processor 235 to generate a user interface. In some embodiments, the user interface module 210 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0066] The user interface module 210 generates a user interface that includes the inpainted image, in some embodiments, the user interface includes options such as editing the inpainted image, sharing the inpainted image, adding the inpainted image to a photo album, and the like.

[0067] Exemplary Flowchart 4 illustrates an example flowchart 400 for generating an inpaint image with bystanders and attached objects removed, according to some embodiments described herein. The method illustrated in flowchart 400 may be performed by computing device 200 of FIG. 2. In some embodiments, flowchart 400 is performed by user device 115, media server 101, or partly by user device 115 and partly by media server 101.

[0068] 4 may begin at block 402. At block 402, bystanders in an initial image are detected. Block 402 may be followed by block 404.

[0069] In block 404, a spectator box is generated that contains the spectator such that all pixels of the spectator lie within the spectator box. Block 404 may be followed by block 406.

[0070] In block 406, a localizer box is generated that encompasses the spectator and one or more objects attached to the spectator. Block 406 may be followed by block 408.

[0071] In block 408, the bystander box and the localizer box are aggregated to form an aggregate box. In some embodiments, the aggregate box is first defined to be the same as the bystander box, and then specific localizer boxes are added based on the application of an IoU calculation. Block 408 may be followed by block 410.

[0072] In block 410, a segmenter is applied to the initial image based on the aggregation box to segment the bystander and one or more objects from the initial image to generate a bystander mask, the bystander mask including a subset of the pixels within the aggregation box. Block 410 may be followed by block 412.

[0073] At block 412, an inpaint image is generated that replaces all pixels in the spectator mask with pixels in the initial image that match the background.

[0074] 5 is an example flowchart 500 for removing bystanders with objects according to some embodiments described herein. The method illustrated in flowchart 500 may be performed by computing device 200 of FIG. 2. In some embodiments, flowchart 500 is performed by user device 115, media server 101, or partly by user device 115 and partly by media server 101.

[0075] The method 500 of Figure 5 may begin at block 502. The method illustrated in flowchart 500 may be performed by the computing device 200 of Figure 2. In some embodiments, flowchart 500 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0076] In block 502, a synthetic training dataset is obtained that includes training images that each include an bystander and a subject attached to an object. In some implementations, the subject in at least a subset of the training images occludes the bystander and / or the attached object. In the case of supervised learning, the training dataset also includes a ground truth bystander mask for each training image. Block 502 may be followed by block 504.

[0077] In block 504, a trained machine learning model is obtained by training a raw machine learning model using the synthetic training dataset as input, and the training is performed using supervised learning. Under supervised learning, the model outputs predicted bystander boxes for the training images. The predicted bystander boxes are compared with ground truth bystander masks corresponding to the training images. Based on the comparison, feedback is provided to adjust one or more parameters of the model under training (raw model). In some implementations, gradient descent may be used to adjust the parameters. This process is performed on a sufficiently large training dataset such that the error between the predicted bystander boxes and the ground truth bystander boxes is reduced. Training may be completed when the error is below a threshold or when the entire training dataset has been utilized. Training may be repeated multiple times with additional training images. Furthermore, retraining may be initiated based on user feedback on the inpaint output images, for example, if the user feedback indicates that the inpaint images are not satisfactory (e.g., do not completely remove bystanders and / or attached objects). The trained model may be made available locally to the user device 115 as part of the media application 103 and / or stored for use on the media server 101. Block 504 may be followed by block 506.

[0078] In block 506, an initial image having an aggregation box surrounding an observer and one or more objects attached to the observer is received as an input to a trained machine learning model. Block 506 may be followed by block 508.

[0079] At block 508, a bystander mask comprising a subset of the pixels within the aggregation box is output using the trained machine learning model.

[0080] 6 illustrates the operation of an exemplary neural network. The neural network machine learning model 600 includes a first layer 604 and a segmentation output layer 608. Although two layers are shown, the neural network machine learning model 600 may include more or fewer layers.

[0081] An initial image 602 is provided as input to an input layer 604. For example, the initial image 602 includes a subject and an bystander attached to one or more objects. The first layer 604 segments the initial image 602 into foreground and background by classifying, for each pixel, the pixel as either part of the foreground or part of the background 606 of the initial image 602.

[0082] The segmentation (606) of the initial image 602 into foreground and background is provided as input to a segmentation output layer 608. The segmentation output layer 608 outputs (610) whether a pixel is part of an observer mask, a subject mask, or neither.

[0083] In addition to the above, the system, program, or functionality described herein may provide the user with controls that allow the user to make choices both regarding when user information (e.g., information regarding the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) may be collected, and when content or communications are sent to the user from the server. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information cannot be identified, or the user's geographic location may be generalized (to city, zip code, state level, etc.) when location information is obtained such that the user's specific location cannot be determined. In this way, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0084] In the above description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the embodiments may be described above primarily with reference to user interfaces and specific hardware. However, the embodiments may be applied to any type of computing device that may receive data and commands, as well as any peripheral device that provides services.

[0085] References herein to "some embodiments" or "some cases" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example may be included in at least one of the implementations described. The appearances of the phrase "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiments.

[0086] Some of the detailed descriptions set forth above have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0087] It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically noted otherwise as will be apparent from the discussion below, throughout this description, discussions utilizing terms including "processing," "operating," "calculating," "determining," or "displaying," etc., will be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that is similarly represented as physical quantities in the computer system's memory, registers, or other such information storage, transmission, or display devices.

[0088] The embodiments herein may also relate to a processor for performing one or more steps of the methods described above. The processor may be a special purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored on non-transitory computer readable storage media including, but not limited to, optical disks, ROM, CD-ROM, magnetic disks, RAM, EPROM, EEPROM, magnetic or optical cards, flash memory including USB keys with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0089] The specification may take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments containing both hardware and software elements, hi some embodiments, the specifications are implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0090] Furthermore, the present specification may take the form of a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this specification, a computer usable or computer readable medium may be any apparatus that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0091] A data processing system suitable for storing or executing program code includes at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory used during the actual execution of the program code, mass storage devices, and cache memories that provide temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from mass storage devices during execution.

Claims

1. 1. A computer-implemented method comprising: Detecting bystanders in an initial image; and generating a spectator box containing the spectator, wherein all pixels of the spectator are within the spectator box, the method further comprising: generating one or more localizer boxes including the bystander box, each of the one or more localizer boxes containing an object attached to the bystander, the method further comprising: aggregating the bystander box and the one or more localizer boxes to form an aggregate box; applying a segmenter to the initial image based on the aggregation box to segment the bystander and one or more of the objects from the initial image to generate a bystander mask for the bystander and one or more of the objects, the bystander mask comprising a subset of pixels within the aggregation box, the subset of pixels encompassing pixels of the bystander and pixels of one or more of the objects, the method further comprising:

11. A method comprising: generating an inpaint image, wherein all pixels in the spectator mask are replaced in the inpaint image with pixels that match a background in the initial image.

2. Prior to applying the segmenter to the initial image, calculating, for each of the one or more localizer boxes, an intersection over union (IoU) ratio of the aggregation box to the localizer box; 2. The method of claim 1 , further comprising: increasing a size of the aggregation box in response to the IoU ratio exceeding a threshold or in response to an overlap area of ​​the aggregation box and the localizer box exceeding an occupancy threshold.

3. The IoU ratio is the overlap area of ​​the aggregation box and the localizer box divided by the union area of ​​the aggregation box and the localizer box, and the method further comprises: The method of claim 2 comprising padding the aggregate box.

4. 4. The method of claim 1, wherein applying the segmenter to the initial image comprises segmenting a foreground from a background to distinguish one or more of the objects pertaining to the bystander from the background.

5. determining whether a subject of the initial image is occluding the bystander; 4. The method of claim 1, further comprising: in response to determining that the subject in the initial image is occluding the bystander, generating a subject mask; and updating the bystander mask to remove pixels in the subject mask in the bystander mask.

6. The method of any one of claims 1 to 3, wherein the segmenter is a trained machine learning model.

7. The one or more localizer boxes include one or more of a subject, an onlooker, or an object in the initial image, and the method further comprises: The method of any one of claims 1 to 3, comprising removing a corresponding localizer box associated with the object.

8. 1. A computing device comprising: A processor; a memory coupled to the processor storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Detecting bystanders in an initial image; and generating an observer box that includes the observer, wherein all pixels of the observer are within the observer box, the operations further comprising: generating one or more localizer boxes including the spectator box, each of the one or more localizer boxes containing an object attached to the spectator, the operation further comprising: aggregating the bystander box and the one or more localizer boxes to form an aggregate box; and applying a segmenter to the initial image based on the aggregation box to segment the bystander and one or more of the objects from the initial image to generate a bystander mask for the bystander and the one or more objects, the bystander mask including a subset of pixels within the aggregation box, the subset of pixels encompassing pixels of the bystander and pixels of one or more of the objects, the operations further comprising: generating an inpaint image, where all pixels in the spectator mask are replaced in the inpaint image with pixels that match a background in the initial image.

9. The operation further comprises: Prior to applying the segmenter to the initial image, calculating, for each of the one or more localizer boxes, an intersection over union (IoU) ratio of the aggregation box to the localizer box; and increasing a size of the aggregation box in response to the IoU ratio exceeding a threshold or in response to an overlap area of ​​the aggregation box and the localizer box exceeding an occupancy threshold.

10. The IoU ratio is the overlap area of ​​the aggregation box and the localizer box divided by the union area of ​​the aggregation box and the localizer box, and the operation further comprises: The computing device of claim 9 , further comprising padding the aggregate box.

11. A computing device according to any one of claims 8 to 10, wherein applying the segmenter to the initial image comprises segmenting a foreground from a background to distinguish one or more of the objects pertaining to the bystander from the background.

12. The operation further comprises: determining whether a subject of the initial image is occluding the bystander; 11. The computing device of claim 8, further comprising: in response to determining that the subject of the initial image is occluding the bystander, generating a subject mask; and updating the bystander mask to remove pixels in the subject mask in the bystander mask.

13. A computing device according to any one of claims 8 to 10, wherein the segmenter is a trained machine learning model.

Images