Disturbance removal in media items based on user input

Abstract

The media application receives user input indicating that one or more objects are to be removed from the media item. The media application converts the user input into a bounding box. The media application provides, based on the bounding box, a crop of the media item to a segmentation machine learning model. The segmentation machine learning model outputs a segmentation mask of one or more segmented objects in the crop of the media item, and a corresponding segmentation score indicating a quality of the segmentation mask.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 257,111, filed October 18, 2021, entitled “Translating User Annotation for Distraction Removal in Media Items,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] The quality of visual media items—such as images (still images, images with selective motion, etc.) and videos—can be improved by removing objects that distract from the focus of the media item or otherwise affect its visual appeal. For example, users sometimes capture pictures or videos that include windmills, people in the background, fences, or other objects that are not part of the main subject the user intends to capture. For instance, a picture might be intended to capture people, trees, buildings, scenery, etc., in the foreground, but there may be one or more distracting objects—in the foreground (e.g., fences, traffic lights, or other objects closer to the camera than the object of interest); in the background (e.g., people in the background, power lines above the object of interest, or other objects farther from the camera than the object of interest); or on the same plane (e.g., a person facing away from the camera but at a similar distance to the object of interest).

[0004] Users can manually remove distracting objects using image or video editing techniques. However, this task can be arduous and incomplete. Furthermore, automatically removing distracting objects is difficult because it may result in false alarms, such as the removal of other objects or parts of objects, or incomplete segmentation leading to the false alarm that parts of removed objects remain visible.

[0005] The background description provided herein is intended to generally present the context of this disclosure. The work of the currently named inventors described in this background section, and descriptions that may not conform to the prior art at the time of filing, are neither expressly nor implied to be considered prior art to this disclosure. Summary of the Invention

[0006] A computer-implemented method includes receiving user input indicating one or more objects to be removed from a media item. The method further includes converting the user input into bounding boxes. The method also includes providing a clipping of the media item to a segmented machine learning model based on the bounding boxes. The method further includes using the segmented machine learning model to output segment masks of one or more segmented objects in the clipping of the media item, and corresponding segment scores indicating the quality of the segment masks.

[0007] In some embodiments, the bounding box is an axis-aligned bounding box or an oriented bounding box. In some embodiments, the user input includes one or more strokes made with reference to the media item. In some embodiments, the bounding box is an oriented bounding box, and wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes. In some embodiments, before providing the clipping of the media item, a segmentation machine learning model is trained using training data including multiple training images and ground-based segmented masks. In some embodiments, the method further includes determining whether the segmented mask is invalid based on one or more of the following: the corresponding segment score fails to meet a threshold score, the number of valid mask pixels is less than a threshold number of pixels, the segmented mask size is less than a threshold size, or the segmented mask is greater than a threshold distance from the region indicated by the user input, and generating a different mask based on the region within the user input in response to determining that the segmented mask is invalid. In some embodiments, the method further includes repairing a portion of the media item that matches the segmented mask to obtain an output media item, wherein one or more objects are not present in the output media item. In some embodiments, the repair is performed using a repair machine learning model, and wherein the media item and the segmented mask are provided as input to the repair machine learning model. In some embodiments, the method further includes providing a user interface that includes the output media item.

[0008] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon causes one or more computers, when executed, to perform operations including: receiving user input indicating one or more objects to be removed from a media item; converting the user input into bounding boxes; providing a clipping of the media item to a segmented machine learning model based on the bounding boxes; and using the segmented machine learning model to output a segment mask of one or more segmented objects in the clipping of the media item and a corresponding segment score indicating the quality of the segment mask.

[0009] In some embodiments, the bounding box is an axis-aligned bounding box or an oriented bounding box. In some embodiments, user input includes one or more strokes made with reference to the media item. In some embodiments, the bounding box is an oriented bounding box, and wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes. In some embodiments, before providing the clipping of the media item, a segmented machine learning model is trained using training data including multiple training images and ground-based segmented masks.

[0010] In some embodiments, the computing device includes one or more processors and memory coupled to the one or more processors, having instructions stored thereon that, when executed by the processors, cause the processors to perform operations. The operations may include receiving user input indicating one or more objects to be removed from a media item, converting the user input into bounding boxes, providing a clipping of the media item to a segmented machine learning model based on the bounding boxes, and using the segmented machine learning model to output segment masks of one or more segmented objects in the clipped media item, along with corresponding segment scores indicating the quality of the segment masks.

[0011] In some embodiments, the bounding box is an axis-aligned bounding box or an oriented bounding box. In some embodiments, the user input includes one or more strokes made with reference to the media item. In some embodiments, the bounding box is an oriented bounding box, and wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes. In some embodiments, prior to providing the clipping of the media item, a segmented machine learning model is trained using training data including multiple training images and ground-based segmented masks. In some embodiments, the operation further includes determining whether the segmented mask is invalid based on one or more of the following: the corresponding segment score fails to meet a threshold score, the number of valid mask pixels is less than a threshold number of pixels, the segmented mask size is less than a threshold size, or the segmented mask is greater than a threshold distance from the region indicated by the user input, and generating a different mask based on the region within the user input in response to determining that the segmented mask is invalid.

[0012] The technology described in the specification advantageously describes a media application that determines user intent associated with user input. For example, when a user circles a portion of an image, the media application determines the specific object the user requests to remove. Attached Figure Description

[0013] Figure 1 This is a block diagram of an example network environment for removing objects from an image according to some embodiments described herein.

[0014] Figure 2 This is a block diagram of an example computing device for removing objects from an image according to some embodiments described herein.

[0015] Figure 3A Example images with user input for removing objects are shown according to some embodiments described herein.

[0016] Figure 3B Example images of axis-aligned bounding boxes according to some embodiments described herein are shown.

[0017] Figure 3C Example images with different segmented masks according to some embodiments described herein are shown.

[0018] Figure 3D Example images showing the removal of objects according to some embodiments described herein are shown.

[0019] Figure 4A Example images of goats with user input to remove fence segments, according to some embodiments described herein, are shown.

[0020] Figure 4B Example images with incorrect bounding boxes are shown according to some embodiments described herein.

[0021] Figure 4C Example images are shown where a goat is removed from a media item according to some embodiments described herein.

[0022] Figure 4D Example images are shown that have oriented bounding boxes that correctly identify fences as objects to be removed, according to some embodiments described herein.

[0023] Figure 4E Example images are shown where fence segments are correctly removed according to some embodiments described herein.

[0024] Figure 5 A flowchart is shown of an example method for generating segmented masks according to some embodiments described herein. Detailed Implementation

[0025] Example Environment 100

[0026] Figure 1 A block diagram of an example environment 100 is shown. In some embodiments, environment 100 includes a media server 101, user device 115a, and user device 115n coupled to network 105. Users 125a and 125n may be associated with corresponding user devices 115a and 115n. In some embodiments, environment 100 may include Figure 1 Other servers or devices not shown. Figure 1 In the other figures, letters following a reference number (e.g., "115a") indicate a reference to the element having that specific reference number. Reference numbers without subsequent letters in the text, such as "115", indicate a general reference to an embodiment of the element with that reference number.

[0027] Media server 101 may include a processor, memory, and network communication hardware. In some embodiments, media server 101 is a hardware server. Media server 101 is communicatively coupled to network 105 via signal line 102. Signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as Wi-Fi. Or other wireless technologies. In some embodiments, media server 101 transmits data to one or more of user devices 115a, 115n and receives data from one or more of user devices 115a, 115n via network 105. Media server 101 may include media application 103a and database 199.

[0028] Database 199 can store machine learning models, training datasets, images, etc. With the user's consent, Database 199 can also store social network data associated with user 125, user preferences, etc.

[0029] User equipment 115 may be a computing device including memory coupled to a hardware processor. For example, user equipment 115 may include a mobile device, tablet computer, mobile phone, wearable device, head-mounted display, mobile email device, portable game player, portable music player, e-reader device, or another electronic device capable of accessing network 105.

[0030] In the illustrated implementation, user equipment 115a is coupled to network 105 via signal line 108, and user equipment 115n is coupled to network 105 via signal line 110. Media application 103 can be stored on user equipment 115a as media application 103b and / or on user equipment 115n as media application 103c. Signal lines 108 and 110 can be wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections, such as Wi-Fi. Or other wireless technologies. User equipment 115a and 115n are accessed by users 125a and 125n, respectively. Figure 1 User equipment 115a and 115n are used as examples. Although Figure 1 Two user equipments 115a and 115n are shown, but the disclosure applies to system architectures with one or more user equipments 115.

[0031] Media application 103 may be stored on media server 101 and / or user device 115. In some embodiments, the operations described herein are performed on media server 101 or user device 115. In some embodiments, some operations may be performed on media server 101 and some on user device 115. The execution of operations conforms to user settings. For example, user 125a may specify that an operation will be performed on the corresponding device 115a instead of on server 101. With such a setting, the operations described herein are performed entirely on user device 115a, without any operation performed on media server 101. Furthermore, user 125a may specify that the user's images and / or other data are stored only on the local user device 115a, and not on media server 101. With such a setting, no user data is transmitted or stored on media server 101. User data is transmitted to media server 101, temporarily or permanently stored by media server 101, and operations performed by media server 101 on such data are performed only if the user has consented to the transmission, storage, and execution of operations by media server 101. Users are given the option to change settings at any time, for example, enabling or disabling the use of media server 101.

[0032] Machine learning models (e.g., neural networks or other types of models), if used for one or more operations, are stored and locally utilized on user device 115 with specific user permission. Server-side models are used only with user permission. Model training is performed using synthetic datasets, as referenced below. Figure 5 Furthermore, a trained model can be provided for use on user device 115. During such use, model training can be performed on the device if permitted by user 125. Updated model parameters can be transmitted to media server 101, for example, with joint learning enabled, if permitted by user 115. The model parameters do not include any user data.

[0033] Media application 103 receives media items. For example, media application 103 receives media items from a camera that is part of user device 115, or media application 103 receives media items via network 105. Media application 103 receives user input indicating one or more objects to be removed from the media item. For example, the user input is a circle around the object to be removed. Media application 103 converts the user input into a bounding box. Media application 103 uses the bounding box to provide a clipping of the media item to a machine learning model. The segmented machine learning model outputs a segmented mask of one or more segmented objects in the clipped media item, along with a corresponding segmented score indicating the quality of the segmented mask. In some embodiments, media application 103 repairs a portion of the media item that matches the segmented mask to obtain an output media item where one or more objects are not present in the output media item.

[0034] In some embodiments, media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a may be implemented using a combination of hardware and software.

[0035] Example computing device 200

[0036] Figure 2 This is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing media application 103a. In another example, the computing device 200 is a user device 115.

[0037] In some embodiments, the computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all of which are coupled via a bus 218. The processor 235 may be coupled to the bus 218 via signal line 222, the memory 237 may be coupled to the bus 218 via signal line 224, the I / O interface 239 may be coupled to the bus 218 via signal line 226, the display 241 may be coupled to the bus 218 via signal line 228, the camera 243 may be coupled to the bus 218 via signal line 230, and the storage device 245 may be coupled to the bus 218 via signal line 232.

[0038] Processor 235 may be one or more processors and / or processing circuitry to execute program code and control the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a system comprising a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing functionality, a dedicated processor for implementing neural network-based processing, neural circuits, processors optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors implementing neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce probabilistic outputs; for example, the output produced by processor 235 may be imprecise or accurate within the expected output range. Processing is not necessarily limited to a specific geographical location or has time constraints. For example, a processor may perform its functions in real-time, offline, batch processing modes, etc. Some processing can be performed at different times and in different locations by different (or the same) processing systems. The computer can be any processor that communicates with memory.

[0039] Memory 237 is disposed in computing device 200 and accessible to processor 235. It can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions executed by the processor or processor group, and positioned separately from and / or integrated with processor 235. Memory 237 may store software executed by processor 235 on computing device 200, including media application 103.

[0040] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, image library applications, image management applications, picture gallery applications, communication applications, web hosting engines or applications, media sharing applications, etc. One or more methods disclosed herein can run in various environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0041] Application data 266 may be data generated by other applications 264 or the hardware of computing device 200. For example, application data 266 may include images used by an image gallery application and user actions identified by other applications 264 (e.g., a social networking application).

[0042] I / O interface 239 can provide functionality that enables computing device 200 to interface with other systems and devices. Interface devices may be included as part of computing device 200 or may be separate from and communicate with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0043] Examples of interface devices capable of connecting to I / O interface 239 may include display 241, which can be used to display content, such as images, videos, and / or user interfaces of output applications as described herein, and to receive touch (or gesture) input from a user. For example, display 241 may be used to display a user interface including graphical guidance on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), a light-emitting diode (LED) or plasma display, a cathode ray tube (CRT), a television, a monitor, a touch screen, a 3D display, or other visual display device. For example, display 241 may be a flat panel display on a mobile device, multiple displays embedded in an eyeglass-shaped or headphone device, or a monitor screen for a computer device.

[0044] Camera 243 can be any type of image capture device capable of capturing media items, including images and / or videos. In some embodiments, camera 243 captures images or videos provided by I / O interface 239 to media application 103.

[0045] Storage device 245 stores data related to media application 103. For example, storage device 245 may store training datasets, which include labeled images, machine learning models, outputs from machine learning models, etc.

[0046] Figure 2 An example of a media application 103 stored in memory 237 is shown, which includes a bounding box module 202, a segmented machine learning module 204, a repairer module 206, and a user interface module 208.

[0047] Bounding box module 202 generates bounding boxes. In some embodiments, bounding box module 202 includes a set of instructions executed by processor 235 to generate bounding boxes. In some embodiments, bounding box module 202 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0048] In some embodiments, the bounding box module 202 receives media items. The media items may be received from the camera 243 of the computing device 200, from application data 266, or from the media server 101 via I / O interface 239. In various embodiments, the media items may be images, videos, a series of images (e.g., GIFs), etc.

[0049] In some implementations, the media item includes user input indicating one or more objects to be removed from the media item. In some implementations, the user input may be received at the client device 110 as touch input via a touchscreen, via a mouse / touchpad / other pointing device, or other suitable input mechanism. In some implementations, the user input is received with reference to a specific media item. In some embodiments, the user input is a manually drawn stroke that surrounds or lies on the object to be erased from the media item. For example, the user input might be a circle around the object, a line or series of lines on top of the object, a square around the object, etc. User input can be provided on the computing device 200 by the user drawing on the touchscreen with a finger or stylus, by inputting with a mouse or pointer, by gesture input (e.g., detected by a camera), etc.

[0050] Go to Figure 3A Example image 300 with user input for removing an object is shown. In this example, the media is an image of a dandelion field with windmills in the background. The user input generally consists of circular shapes 305, 310, and 315 surrounding the object to be removed. User input 305 surrounds the first windmill, user input 310 surrounds the two windmills, and user input 315 surrounds the fourth windmill.

[0051] In some embodiments, the bounding box module 202 converts user input into a bounding box. The bounding box module 202 identifies the object associated with the user input. For example, in Figure 3AIn this context, the bounding box module 202 identifies the association between user input 305 and the windmills enclosed by user input 305. In some embodiments, where the user input may include multiple objects, the bounding box module 202 identifies the percentage of objects associated with the user input. For example, user input 310 encloses almost all pixels of an image corresponding to two windmills. As a result, the bounding box module 202 associates user input 310 with both windmills. In some embodiments, where the user input does not enclose all objects, the bounding box module 202 determines whether the amount of user input associated with an object exceeds a threshold percentage of the object (e.g., measured in pixels). For example, user input 315 includes the entire windmill except for one of the blades, and the percentage is 85%, which exceeds the 70% threshold percentage.

[0052] In some embodiments, the bounding box module 202 identifies an object associated with the user input and compares the object's identity with a list of objects typically removed to determine whether the user input includes a specific object. For example, the list of typically removed objects might include people, power lines, scooters, trash cans, etc. If the user input simultaneously surrounds a person and part of a tree in the background, the bounding box module 202 can determine that the user input corresponds to a person rather than a tree, because only people, not trees, are part of the list of typically removed objects.

[0053] Bounding box module 202 generates bounding boxes that include one or more objects. In some embodiments, the bounding box is a rectangular bounding box that contains all the pixels of one or more objects. In some embodiments, bounding box module 202 uses a suitable machine learning algorithm, such as a neural network or more specifically, a convolutional neural network, to identify one or more objects and generate bounding boxes. The bounding boxes are associated with the x and y coordinates of the media item (image or video).

[0054] In some embodiments, the bounding box module 202 converts user input into an axis-aligned bounding box or a directional bounding box. The axis-aligned bounding box is aligned with the x and y axes of the media item. In some embodiments, the axis-aligned bounding box fits snugly around the stroke, such that the edges of the bounding box contact the widest part of the stroke. The axis-aligned bounding box is the smallest box that includes the object indicated by the user input. Go to Figure 3B Example image 310 with axis-aligned bounding boxes is shown. Bounding boxes 325, 330, and 335 each include one or more corresponding objects, and bounding boxes 325, 330, and 335 enclose corresponding user input strokes.

[0055] exist Figure 3BIn this implementation, the three strokes input by the user are converted into three bounding boxes, but other embodiments are also possible, such as four bounding boxes, where each bounding box corresponds to a corresponding object and fits tightly around the stroke, except for areas where multiple objects are separated. For example, bounding box 330 can be divided into two boxes, where the outermost line of the stroke is aligned with the bounding box, and there is one or more additional lines in the center to indicate separation between objects.

[0056] In some embodiments, the bounding box module 202 generates an oriented bounding box whose orientation matches the orientation of the stroke. For example, when user input is in one direction, such as when a user provides one or more lines on a media item, the oriented bounding box may be applied by the bounding box module 202. In some embodiments, the bounding box module 202 generates an oriented bounding box that fits snugly around a stroke that can be rotated relative to the image axis. In some embodiments, the oriented bounding box is any bounding box, wherein the faces and edges of the bounding box are not parallel to the edges of the media item.

[0057] In some embodiments, the bounding box module 202 generates a clipping of the bounding box based on the bounding box. For example, the bounding box module 202 generates a clipping that uses the coordinates of the bounding box to generate a clipping that includes one or more objects within the bounding box.

[0058] In some embodiments, the segmented machine learning module 204 includes (and optionally also performs training on) a trained model, referred to herein as a segmented machine learning model. In some embodiments, the segmented machine learning module 204 is configured to apply the machine learning model to input data, such as application data 266 (e.g., media items captured by user device 115), and output a segmented mask. In some embodiments, the segmented machine learning module 204 may include code to be executed by processor 235. In some embodiments, the segmented machine learning module 204 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0059] In some embodiments, the segmented machine learning module 204 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) enabling the processor 235 to apply a machine learning model. In some embodiments, the segmented machine learning module 204 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the segmented machine learning module 204 may provide an application programming interface (API) that can be used by the operating system 262 and / or other applications 264 to invoke the segmented machine learning module 204, for example, to apply a machine learning model to application data 266 to output a segmented mask.

[0060] The segmented machine learning module 204 uses training data to generate a trained segmented machine learning model. For example, the training data may include training images and ground truth segmented masks. The training images may be cropped bounding boxes from manually segmented images and / or cropped bounding boxes from synthetic images. In some embodiments, the segmented machine learning module 204 uses axis-aligned bounding boxes or oriented bounding boxes to train the segmented machine learning model.

[0061] In some embodiments, training data may include synthetic data generated for training purposes, such as data not based on activities in the training context, for example, data generated from simulated or computer-generated images / videos. Training data may include synthetic images cropped from synthetic image bounding boxes. In some embodiments, the synthetic image is generated by overlaying a two-dimensional or three-dimensional object onto a background image. Three-dimensional objects can be rendered from a specific view to convert three-dimensional objects into two-dimensional objects.

[0062] Training data can be obtained from any source, such as a data repository specifically tagged for training, data licensed for use as training data for machine learning, etc. In some embodiments, training may occur on media server 101, which provides training data directly to user device 115, training may occur locally on user device 115, or a combination of both.

[0063] In some embodiments, the segmented machine learning module 204 uses weights taken from another application and without editing / transfer. For example, in these embodiments, the trained model may be generated, for instance, on different devices and provided as part of media application 103. In various embodiments, the trained model may be provided as a data file including the model structure or form (e.g., defining the number and type of neural network nodes, the connectivity between nodes, and organizing the nodes into multiple layers) and associated weights. The segmented machine learning module 204 can read the data file of the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0064] A trained machine learning model can include one or more model forms or structures. For example, a model form or structure can include any type of neural network, such as linear networks, deep learning neural networks that implement multiple layers (e.g., "hidden layers" between the input and output layers, each of which is a linear network), convolutional neural networks (e.g., networks that split or divide input data into multiple parts or tiles, process each tile individually using one or more neural network layers, and aggregate the processing results for each tile), sequence-to-sequence neural networks (e.g., networks that receive sequential data as input—such as words in a sentence, frames in a video, etc.—and produce a sequence of results as output), and so on.

[0065] The model form or structure can specify the connectivity between various nodes and the organization of nodes within layers. For example, nodes in the first layer (e.g., the input layer) can receive data as input data or application data. Such data may include, for example, one or more pixels for each node, e.g., when the trained model is used for analysis, for example, of an initial image. Subsequent intermediate layers can receive the outputs of nodes in the previous layer as input, according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer may output segments between the foreground and background. The final layer (e.g., the output layer) produces the output of the machine learning model. For example, the output layer may receive segments of the initial image to the foreground and background and output whether a pixel is part of a segmented mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0066] In various embodiments, the trained model may include one or more models. One or more models may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, a node may be a computational node without memory, for example, configured to process a unit of input to produce a unit of output. Computations performed by a node may include, for example, multiplying each of the multiple node inputs by a weight to obtain a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce a node output. In some embodiments, the computation performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, computations are performed in parallel by multiple nodes, for example, using multiple processor cores of a multi-core processor, a single processing unit using a graphics processing unit (GPU), or a dedicated neural circuit. In some embodiments, a node may include memory, for example, to store and use one or more earlier inputs while processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. LSTM nodes can use memory to maintain a "state," which allows the node to operate like a finite state machine (FSM).

[0067] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may be started as multiple nodes organized into layers specified by a model form or structure. During initialization, appropriate weights may be applied to the connections between each pair of nodes connected according to the model form, such as nodes in consecutive layers of a neural network. For example, the appropriate weights may be randomly assigned or initialized to default values. The model can then be trained, for example, using training data to produce results.

[0068] Training can include the application of supervised learning techniques. In supervised learning, training data can include multiple inputs (e.g., manually annotated clips and synthesized media items) and a corresponding ground truth output for each input (e.g., a ground truth segmentation mask that correctly identifies one or more objects to be removed from each stroke of a media item). Based on a comparison between the model output and the ground truth output, weights are automatically adjusted, for example, to increase the probability that the model will generate ground truth outputs for the media items.

[0069] In some embodiments, during training, the segmented machine learning module 204 outputs a segmented mask and a segmented score, which indicates the quality of the segmented mask that identifies objects to be erased in a media item. The segmented score may reflect the intersection-over-union (IoU) ratio between the segmented mask output by the segmented machine learning model and the ground truth segmented mask.

[0070] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights, for example, downloaded from a server providing the weights. In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmented machine learning module 204 may generate a trained model based on previous training, for example, by the developer of the segmented machine learning module 204, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights, for example, downloaded from a server providing the weights.

[0071] In some embodiments, the segmented machine learning module 204 receives clippings of media items. The segmented machine learning module 204 provides the clippings of media items as input to a trained machine learning model. In some embodiments, the trained machine learning model outputs segment masks of one or more segmented objects in the clippings of the media items and corresponding segment scores indicating the quality of the segment masks. In some embodiments, the segment scores are based on segment scores generated during the training of the machine learning model, reflecting the IoU between the segment masks output by the machine learning model and the ground truth segment masks. In some embodiments, the segment scores are numbers in the total, such as 40 / 100. Other representations of the segment scores are possible.

[0072] In some embodiments, the segmented machine learning model outputs a confidence value for each segment mask output by the trained machine learning model. The confidence value can be expressed as a percentage, a number from 0 to 1, etc. For example, the machine learning model outputs an 85% confidence value that the segment mask correctly covers the object identified in the user input.

[0073] In some embodiments, the segmented machine learning module 204 determines that a segmented mask has not been successfully generated. For example, the segment score may not meet a threshold score. In another example, the segmented machine learning module 204 may determine the number of valid mask pixels and determine that this number is below a threshold number of pixels. In another example, the segmented machine learning module 204 may determine the size of the segmented mask and that the size of the segmented mask is below a threshold size. In yet another example, the segmented machine learning module 204 may determine the distance between the segmented mask and the region indicated by the user input, and that distance is greater than a threshold distance. In one or more of these instances, the segmented machine learning module 204 outputs different segmented masks based on the region within the user input.

[0074] Go to Figure 3C Example image 340 is shown using different segmented masks 345, 350, and 355. In this example, the segmented machine learning module 204 outputs different segmented masks, which include pixels corresponding to the regions within the user input.

[0075] Repair module 206 generates output media items where one or more objects are absent (erased from the source media item). In some embodiments, repair module 206 includes a set of instructions executable by processor 235 to generate the output media items. In some embodiments, repair module 206 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0076] In some embodiments, the repair module 206 receives a segmented mask from the segmented machine learning module 204. The repair module 206 performs repair on a portion of a media item that matches the segmented mask. For example, the repair module 206 replaces pixels within the segmented mask with pixels that match the background in the media item. In some embodiments, the pixels that match the background may be based on another media item at the same location. Figure 3D This example demonstrates the repair of image 360, where objects disappear from the output media items after the repair.

[0077] In some embodiments, the repair module 206 trains a repair machine learning model to receive media items and segmented masks from the segmented machine learning module 204 as input, and outputs an output media item having one or more objects that are not present in the output media item.

[0078] User interface module 208 generates a user interface. In some embodiments, user interface module 208 includes a set of instructions executable by processor 235 to generate the user interface. In some embodiments, user interface module 208 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0079] User interface module 208 generates the user interface, which is executed by... Figure 2 Any steps performed by the modules in and Figure 5 Before proceeding with the steps described, request permission from the user to access their media items.

[0080] User interface module 208 generates a user interface that includes media items and accepts user input to identify one or more objects to be removed. For example, the user interface accepts touch input via strokes. User input is an instruction from the user to remove distracting (or other problematic) objects from the media items. For example, the media item might be an image of a family on a beach, and the distracting objects might be two people walking along the edge of the beach in the background. The user can use the user interface to circle the two people walking along the edge of the beach.

[0081] User interface module 208 generates a user interface that includes the repaired output media item. Continuing the example, the media item is a family on a beach, with no two people walking along the edge of the beach. In some embodiments, the output media item may be labeled (visually) or marked (in code, e.g., steganographically) to indicate that the media item is being edited to erase one or more objects. In some embodiments, the user interface includes options for editing the output media item, sharing the output media item, adding the output media item to an album, etc. Options for editing the output media item may include the ability to undo object erasure.

[0082] In some embodiments, output media items may be labeled (visually) or marked (in code, for example, shorthand) to indicate that the media item is edited to erase one or more objects.

[0083] In some embodiments, the user interface module 208 receives feedback from a user on the user device 115. The feedback may take the form of the user posting output media items, deleting output media items, sharing output media items, etc.

[0084] Example Oriented Bounding Box

[0085] Figure 4A An example image 400 of a goat with user input to remove fence segments, according to some embodiments described herein, is shown. A bounding box module 202 receives the user input and generates an oriented bounding box. The orientation of the oriented bounding box is determined based on the orientation input by the user. Figure 4A In the text, user input 405 is a stroke along the diagonal of the chain link fence. Bounding box 407 is an axis-aligned bounding box.

[0086] Figure 4B Example image 410 showing an incorrect bounding box according to some embodiments described herein. Because the axis-aligned bounding box is a rectangular box with its sides aligned with the x-axis and y-axis, the bounding box incorrectly identifies the goat, instead of the chain link fence identified by the user input as the object to be removed, as the object to be removed.

[0087] Figure 4C Example image 420 showing the removal of the goat from a media item is shown.

[0088] Figure 4D Example image 430 is shown, where the bounding box module 202 uses a directional bounding box, which correctly identifies the fence as the object to be removed. Figure 4D As shown, when the segmented machine learning module 204 receives a clipped version of the oriented bounding box, the resulting segmented mask is closer to capturing the user intent to remove a portion of the chain link fence than when the segmented machine learning module 204 receives a clipped version of the axis-aligned bounding box, which incorrectly interprets the user intent as selecting the goat behind the chain link fence.

[0089] Figure 4E Example image 440 shows a fence segment being correctly removed according to some embodiments described herein.

[0090] Example method 500

[0091] Figure 5 A flowchart of example method 500 for generating segmented masks is shown. Figure 5 Method 500 can begin from box 502. Method 500 shown in the flowchart can be derived from... Figure 2 The method 500 is executed by the computing device 200. In some embodiments, the method 500 is executed by the user device 115, the media server 101, or partly on the user device 115 and partly on the media server 101.

[0092] At box 502, user permission is received to implement method 500. For example, a user can load the application to provide user input by circling an object in a media item, but before displaying the media item, the user interface requests user permission to access the media item associated with the user. The user interface can also request modification of the media item's permission so that the user is only allowed to access specific media items, ensuring that no media item is stored or transferred to the server without the user's permission, etc. Box 502 can be followed by box 504.

[0093] At box 504, it is determined whether user permission has been received. If user permission has not been received, box 506 follows box 504, which stops method 500. If user permission has been received, box 508 follows box 504.

[0094] At box 508, user input is received, indicating one or more objects to be removed from the media item. For example, the image might include a trash can in the background, and the user input might be a circle around the trash can. Box 508 may be followed by box 510.

[0095] At box 510, the user input is converted into a bounding box. For example, the bounding box could be an axis-aligned bounding box or an oriented bounding box. Box 510 may be followed by box 512.

[0096] At box 512, media items are clipped to the segmented machine learning model based on bounding boxes. Box 506 can be followed by box 514.

[0097] At box 514, using a trained machine learning model, the segment mask of one or more segmented objects in the clipping of the media item is output, along with the corresponding segment score indicating the quality of the segment mask.

[0098] Following the foregoing description, users may be given control over whether and when the systems, programs, or functions described herein can collect user information (e.g., information about the user's media items, including images and / or videos, social networks, social actions or activities, occupation, user preferences (e.g., relative to objects in an image) or the user's current location), and whether the user sends content or communications from the server. Furthermore, some data may be processed in one or more ways before storage or use to remove personally identifiable information. For example, a user's identity may be processed to the point that it is impossible to determine personally identifiable information for the user, or, where location information (such as city, zip code, or state level) is available, the user's geographic location may be generalized to the point that it is impossible to determine the user's specific location. Therefore, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0099] In the foregoing description, numerous specific details have been set forth for illustrative purposes in order to provide a thorough understanding of the specification. However, it will be apparent to those skilled in the art that the disclosure may be practiced without these specific details. In some cases, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the above embodiments may be described primarily with reference to a user interface and specific hardware. However, this embodiment can be applied to any type of computing device capable of receiving data and commands, as well as any peripheral device providing services.

[0100] The phrase "some embodiments" or "some examples" mentioned in the specification refers to specific features, structures, or characteristics described in relation to said embodiments or examples that may be included in at least one of the described implementations. The phrase "in some embodiments" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0101] Certain parts of the above detailed description are presented in the form of algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to those skilled in the art. An algorithm here is generally considered to be a self-consistent sequence of steps that leads to a desired result. Steps are those steps that require physical operations on physical quantities. Typically, though not always, these quantities take the form of electrical or magnetic data capable of being stored, transmitted, combined, compared, and otherwise manipulated. It has proven convenient, sometimes, primarily for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0102] However, it should be remembered that all these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities. Unless it becomes apparent from the discussion below that a specific statement is made otherwise, it will be understood that throughout the description, the use of terms such as “processing” or “calculating” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the computer system registers and memory and convert that data into other data similarly represented as physical quantities within the computer system memory or registers or other such information storage, transmission, or display devices.

[0103] The embodiments described in the specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor, selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including optical discs, ROMs, CD-ROMs, magnetic disks, RAM, EPROMs, EEPROMs, magnetic cards or optical cards, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0104] This specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments that include both hardware and software elements. In some embodiments, the specification is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0105] Furthermore, the description may take the form of a computer program product, accessible from a computer-usable or computer-readable medium, providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any means that may include, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0106] A data processing system suitable for storing or executing program code will include at least one processor that is directly or indirectly coupled to memory elements via a system bus. Memory elements may include local memory, mass storage, and cache memory used during the actual execution of the program code, which provide temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution.

Claims

1. A computer-implemented method, comprising: Receive user input indicating one or more objects to be removed from a media item; Convert the user input into a bounding box; The media items are clipped to the segmented machine learning model based on the bounding box. The segmented machine learning model is used to output the segmented mask of one or more segmented objects in the cropping of the media item, and the corresponding segmented score indicating the quality of the segmented mask; The segmented mask is determined to be invalid based on one or more of the following: the corresponding segment score fails to meet the threshold score, the number of valid mask pixels is lower than the threshold number of pixels, the segmented mask size is lower than the threshold size, or the segmented mask is greater than the threshold distance from the area indicated by the user input. as well as In response to determining that the segmented mask is invalid, a different mask is generated based on the region within the user input.

2. The method according to claim 1, wherein, The bounding box is an axis-aligned bounding box or an oriented bounding box.

3. The method according to claim 1, wherein, The user input includes one or more strokes made with reference to the media item.

4. The method according to claim 3, wherein, The bounding box is an oriented bounding box, wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes.

5. The method according to claim 1, wherein, Before providing the cropped media item, the segmented machine learning model is trained using training data that includes multiple training images and ground-based segmented masks.

6. The method according to claim 1, further comprising: A user interface is provided for receiving the user input, wherein the user input is selected from the group consisting of: a circle around the one or more objects, one or more lines on top of the one or more objects, a square around the one or more objects, and combinations thereof.

7. The method of claim 1, further comprising repairing a portion of the media item matching the different mask to obtain an output media item, wherein, One or more of the objects do not exist in the output media item.

8. The method according to claim 7, wherein, The repair is performed using a repair machine learning model, wherein the media item and the different mask are provided as input to the repair machine learning model.

9. The method of claim 7, further comprising providing a user interface including the output media item.

10. A non-transitory computer-readable medium storing instructions thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform operations, the operations including: Receive user input indicating one or more objects to be removed from a media item; Convert the user input into a bounding box; The media items are clipped to the segmented machine learning model based on the bounding box. The segmented machine learning model is used to output the segmented mask of one or more segmented objects in the cropping of the media item, and the corresponding segmented score indicating the quality of the segmented mask; The segmented mask is determined to be invalid based on one or more of the following: the corresponding segment score fails to meet the threshold score, the number of valid mask pixels is lower than the threshold number of pixels, the segmented mask size is lower than the threshold size, or the segmented mask is greater than the threshold distance from the area indicated by the user input. as well as In response to determining that the segmented mask is invalid, a different mask is generated based on the region within the user input.

11. The computer-readable medium of claim 10, wherein, The bounding box is an axis-aligned bounding box or an oriented bounding box.

12. The computer-readable medium of claim 10, wherein, The user input includes one or more strokes made with reference to the media item.

13. The computer-readable medium according to claim 12, wherein, The bounding box is an oriented bounding box, wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes.

14. The computer-readable medium of claim 10, wherein, Prior to this provision, the segmented machine learning model was trained using training data that included multiple training images and ground-based segmented masks.

15. A computing device, comprising: processor; as well as A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Receive user input indicating one or more objects to be removed from a media item; Convert the user input into a bounding box; The media items are clipped to the segmented machine learning model based on the bounding box. The segmented machine learning model is used to output the segmented mask of one or more segmented objects in the cropping of the media item, and the corresponding segmented score indicating the quality of the segmented mask; The segmented mask is determined to be invalid based on one or more of the following: the corresponding segment score fails to meet the threshold score, the number of valid mask pixels is lower than the threshold number of pixels, the segmented mask size is lower than the threshold size, or the segmented mask is greater than the threshold distance from the area indicated by the user input; and In response to determining that the segmented mask is invalid, a different mask is generated based on the region within the user input.

16. The computing device according to claim 15, wherein, The bounding box is an axis-aligned bounding box or an oriented bounding box.

17. The computing device according to claim 15, wherein, The user input includes one or more strokes made with reference to the media item.

18. The computing device according to claim 17, wherein, The bounding box is an oriented bounding box, wherein the orientation of the oriented bounding box matches the orientation of at least one of the one or more strokes.

19. The computing device according to claim 15, wherein, Before providing the cropped media item, the segmented machine learning model is trained using training data that includes multiple training images and ground-based segmented masks.

20. The computing device according to claim 15, wherein, The operation also includes: A user interface is provided for receiving the user input, wherein the user input is selected from the group consisting of: a circle around the one or more objects, one or more lines on top of the one or more objects, a square around the one or more objects, and combinations thereof.

Images