Subdividing and removing objects from media items

Abstract

A media application generates training data including a first set of media items and a second set of media items, the first set of media items including a distracting object corresponding to the second set of media items and being manually segmented, and trains a segmentation machine learning model based on the training data to receive media items having one or more distracting objects and to output segmentation masks for one or more segmented objects corresponding to the one or more distracting objects.

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 257,114 (filed October 18, 2021, entitled "Segmenting and Removing Objects from Media Items"), which is incorporated by reference in its entirety herein. [Background technology]

[0002] background The user-perceived quality of visual media items such as images (static images, images with selective motion, etc.) and videos can be improved by removing certain objects that disrupt the focus of the media item. While the distracting objects can be removed by hand, the task can be difficult and incomplete. Furthermore, distracting objects are difficult to automatically remove from media items because such removal can result in images in which additional objects or parts of objects are also removed (falsely identified as distracting objects), overly triggered and unrealistic results if additional objects are removed, or incomplete subdivisions in which parts of the removed objects are still visible. Summary of the Invention [Problem to be solved by the invention]

[0003] Users can use manual image or video editing techniques to remove distracting objects. However, this task can be tedious and incomplete. Furthermore, automatically removing distracting objects is difficult because it can result in false positives that additional objects or parts of objects are also removed, or because imperfect segmentation can result in parts of the removed object still being visible.

[0004] The discussion of the background art provided herein is for purposes of generally presenting the context of the present disclosure. The work of the presently named inventors is not expressly or impliedly admitted as prior art to the present disclosure, as are aspects of the description that may not qualify as prior art at the time of filing, to the extent that they are described in this background art section. [Means for solving the problem]

[0005] overview A computer-implemented method includes generating training data including a first set of media items and a second set of media items, the first set of media items corresponding to the second set of media items and including a distracting object that is manually segmented, the method further includes training a segmentation machine learning model based on the training data to receive a media item having one or more distracting objects and output a segmentation mask for one or more segmented objects corresponding to the one or more distracting objects.

[0006] In some embodiments, the one or more distracting objects are power lines, and the method further comprises identifying one or more media items from a first set of media items comprising one or more broken power lines, and augmenting the one or more media items to compensate for the one or more broken power lines in the training data. In some embodiments, augmenting the one or more media items to compensate for the one or more broken power lines comprises modifying a local contrast in the one or more media items and blending a portion of the one or more media items with a portion of the one or more corresponding augmented media items to generate a corresponding one or more augmented media items. In some embodiments, the local contrast is modified using a gain curve that sums two bias curves. In some embodiments, generating the training data comprises augmenting the one or more media items by applying augmentation to a segmentation mask of the one or more distracting objects. In some embodiments, the one or more distracting objects are organized into categories, the categories comprising at least one selected from the group of power lines, utility poles, towers, and combinations thereof. In some embodiments, training the segmentation machine learning model comprises generating a large-scale machine learning model based on the training data, and refining the large-scale machine learning model into a trained segmentation machine learning model by performing inference on the training data segmented by the large-scale machine learning model. In some embodiments, the training data further comprises a synthetic image in which the distracting object is added in front of an outdoor environment object.

[0007] In some embodiments, a computer-implemented method for removing distracting objects from a media item includes receiving a media item from a user; identifying one or more distracting objects within the media item; providing the media item to a trained segmentation machine learning model; using the trained segmentation machine learning model to output a segmentation mask for the one or more distracting objects in the media item; and repairing portions of the media item that match the segmentation mask to obtain an output media item, wherein the one or more distracting objects are absent from the output media item.

[0008] The one or more distracting objects are organized into categories, the categories including at least one selected from the group of power lines, utility poles, towers, and combinations thereof. In some embodiments, the method further includes providing a user with suggestions for removing the one or more distracting objects from the media items. In some embodiments, the trained segmentation machine learning model is trained using training data including a first set of media items and a second set of media items, the first set of media items including a distracting object that corresponds to the second set of media items and is manually segmented.

[0009] In some embodiments, a non-transitory computer-readable medium has instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform a plurality of operations, the plurality of operations including generating training data including a first set of media items and a second set of media items, the first set of media items corresponding to the second set of media items and including a distracting object that is manually segmented, the plurality of operations further including receiving a media item having one or more distracting objects and training a segmentation machine learning model based on the training data to output a segmentation mask for one or more segmented objects corresponding to the one or more distracting objects.

[0010] In some embodiments, the one or more distracting objects are power lines, and the operations further include identifying one or more media items from a first set of media items including one or more broken power lines, and augmenting the one or more media items to compensate for the one or more broken power lines in the training data. In some embodiments, augmenting the one or more media items to compensate for the one or more broken power lines includes modifying a local contrast in the one or more media items and blending a portion of the one or more media items with a corresponding portion of the one or more augmented media items to generate a corresponding one or more augmented media items. In some embodiments, the local contrast is modified using a gain curve that sums two bias curves. In some embodiments, generating the training data includes augmenting the one or more media items by applying augmentation to a segmentation mask of the one or more distracting objects. In some embodiments, the one or more distracting objects are organized into categories, the categories including at least one selected from the group of power lines, utility poles, towers, and combinations thereof. In some embodiments, training the segmentation machine learning model comprises generating a large-scale machine learning model based on the training data, and refining the large-scale machine learning model into a trained segmentation machine learning model by performing inference on the training data segmented by the large-scale machine learning model. In some embodiments, the training data further comprises a synthetic image in which the distracting object is added in front of an outdoor environment object.

[0011] The application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawings will be provided by the Office upon request and payment of the necessary fee. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram of an example network environment for removing distracting objects from media items according to some embodiments described herein. [Diagram 2] 1 is a block diagram of an example computing device for removing distracting objects from a media item according to some embodiments described herein. [Diagram 3] FIG. 1 illustrates an example of generating synthetic images for training a segmentation machine learning model, according to some embodiments described herein. [Figure 4] 1A-1C are diagrams illustrating example images with different local contrasts, in accordance with certain embodiments described herein. [Diagram 5] 1A-1C are diagrams illustrating example original, augmented, and blended images in accordance with certain embodiments described herein. [Figure 6] 1 is an example of a bias curve and how it affects image contrast according to some embodiments described herein. [Figure 7] 1A-1C are diagrams illustrating an example original image and an example image with increased local contrast, according to some embodiments described herein. [Figure 8] FIG. 2 illustrates an example of contrast between different objects as a function of bias curve, according to some embodiments described herein. [Figure 9] 1A-1C are diagrams illustrating example images with varying brightness, consistent with certain embodiments described herein. [Figure 10] 5A-5C are diagrams illustrating examples of images affected by different bias curves, in accordance with certain embodiments described herein. [Figure 11]11A-11C are diagrams illustrating modifications to a segmentation mask based on different dilations in accordance with certain embodiments described herein. [Figure 12] 1A-1C are diagrams illustrating an example original image and an example image with local blurring and an expanded segmentation mask, according to some embodiments described herein. [Figure 13] 1 is a flowchart of an example method for training a segmentation machine learning model according to some embodiments described herein. [Figure 14] 1 is a flowchart of an example method for removing distracting objects from a media item according to some embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Detailed Description Exemplary Environment 100 FIG. 1 shows a block diagram of an exemplary environment 100. In some embodiments, the environment 100 includes a media server 101, a user device 115a, and a user device 115n coupled to a network 105. The users 125a, 125n may be associated with the respective user devices 115a, 115n. In some embodiments, the environment 100 may include other servers or devices not shown in FIG. 1. In FIG. 1 and the remaining figures, a letter following a reference number, e.g., "115a," represents a reference to the element with that particular reference number. A reference number in text without a following letter, e.g., "115," represents a general reference to an embodiment of the element with that reference number.

[0014] The media server 101 may include a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively connected to a network 105 via signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technology. In some embodiments, the media server 101 transmits data to or receives data from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0015] The database 199 may store machine learning models, training data sets, images, etc. Upon receiving user consent, the database 199 may store social network data associated with the user 125, user preferences of the user 125, etc.

[0016] The user device 115 may be a computing device that includes a memory coupled to a hardware processor. For example, the user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing the network 105.

[0017] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored as media application 103b on user device 115a and / or media application 103c on user device 115n. Signal lines 108 and 110 may be wired connections such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections such as Wi-Fi, Bluetooth, or other wireless technologies, etc. User devices 115a, 115n are accessed by users 125a, 125n, respectively. User devices 115a, 115n in FIG. 1 are used as an example. Although FIG. 1 shows two user devices 115a and 115n, the present disclosure applies to a system architecture having one or more user devices 115.

[0018] The media application 103 may be stored on the media server 101 and / or the user device 115. In some embodiments, the operations described herein are executed on the media server 101 or the user device 115. In some embodiments, some operations may be executed on the media server 101 and some operations may be executed on the user device 115. The execution of the operations is subject to user settings. For example, the user 125a may specify settings that operations should be executed on the respective device 115a and not on the media server 101. In such settings, the operations described herein are executed entirely on the user device 115a and not on the media server 101. Additionally, the user 125a may specify that the user's images and / or other data are stored locally only on the user device 115a and not on the media server 101. In such settings, the user data is not transmitted to or stored in the media server 101. The transmission of user data to the media server 101, the temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 are performed only if the user consents to the transmission, storage, and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example to enable or disable the use of the media server 101.

[0019] Machine learning models (e.g., neural networks or other types of models) are stored locally on the user device 115 and utilized with specific user authorization when utilized for one or more operations. Server-side models are utilized only if authorized by the user. Model training is performed using a synthesized dataset, as described below with reference to FIG. 5. Furthermore, trained models may be provided for use on the user device 115. During such use, on-device training of the model may be performed, if authorized by the user 125. Updated model parameters may be transmitted to the media server 101, for example, to enable federated learning, if authorized by the user 125. Model parameters do not include any user data.

[0020] The media application 103 trains a segmentation machine learning model to remove one or more distracting objects from media items by generating training data including a first set of media items and a second set of media items, the first set of media items corresponding to the second set of media items and including a distracting object that is manually segmented. As used herein, a distracting object is an object that falls into a predefined category of objects that are frequently undesirable in an image, and the techniques presented herein allow for the "distracting object" to be seamlessly removed from an image by replacing it with its background. An example of a distracting object used for illustration purposes herein is a power line, a pole, or a tower. The media application 103 may train the segmentation machine learning model based on the training data to receive a media item having one or more distracting objects and output a segmentation mask for one or more segmented objects corresponding to the one or more distracting objects.

[0021] In some embodiments, the media application 103 may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / co-processor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using a combination of hardware and software.

[0022] Exemplary Computing Device 200 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. The computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 used to implement a media application 103a. In another example, the computing device 200 is a user device 115.

[0023] In some embodiments, computing device 200 includes a processor 235, memory 237, input / output (I / O) interface 239, a display 241, a camera 243, and storage 245, all coupled via a bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage 245 may be coupled to bus 218 via signal line 232.

[0024] The processor 235 may be one or more processors and / or processing circuits for executing program code and controlling basic operations of the computing device 200. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), complex programmable logic devices (CPLDs), systems having dedicated circuits for achieving functionality, dedicated processors for implementing neural network model-based processing, neural circuits, processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 235 may include one or more co-processors for implementing neural network processing. In some embodiments, the processor 235 may be a processor that processes data to generate a probabilistic output, e.g., the output generated by the processor 235 may be inaccurate or may be accurate within a range from an expected output. The processing need not be limited to a particular geographic location, nor need it have time limitations. For example, the processor may perform its functions in real-time, offline, in batch mode, etc. Parts of the processing may be performed by different (or the same) processing systems at different times, in different locations, etc. A computer may be any processor in communication with a memory.

[0025] Memory 237 is provided within computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read only memory (ROM), electrically erasable read only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor or set of processors, and may be located separately from and / or integrated with processor 235. Memory 237 may store software operated on computing device 200 by processor 235, including media application 103.

[0026] Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, for example, as a standalone computer program that may run on any type of computing device, as a web application having a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0027] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions identified by other applications 264 (e.g., social networking applications), and the like.

[0028] The I / O interface 239 can provide functionality that allows the computing device 200 to interface with other systems and devices. The interfaced devices can be included as part of the computing device 200 or can be separate and in communication with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage 245), and input / output devices can communicate through the I / O interface 239. In some embodiments, the I / O interface 239 can connect to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0029] Some examples of interface connection devices that can be connected to I / O interface 239 may include a display 241 that may be used to display content, e.g., images, videos, and / or user interfaces of output applications as described herein, and to receive touch (or gesture) input from a user. For example, display 241 may be utilized to display a user interface including a graphical guide on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, display 241 may be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen for a computing device.

[0030] The camera 243 may be any type of image capture device capable of capturing media items, including images and / or video. In some embodiments, the camera 243 captures images or video that the I / O interface 239 provides to the media application 103.

[0031] Storage 245 stores data related to media application 103. For example, storage 245 can store training data sets that include labeled images, machine learning models, output from the machine learning models, and the like.

[0032] FIG. 2 illustrates an exemplary media application 103 stored in memory 237 that includes an object detection module 202 , a segmentation module 204 , a repair module 206 , and a user interface module 208 .

[0033] The object detection module 202 detects distracting objects of a media item. In some embodiments, the object detection module 202 includes a set of instructions executable by the processor 235 to detect distracting objects of a media item. In some embodiments, the object detection module 202 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0034] In some embodiments, the object detection module 202 receives a media item, which may be received from the camera 243 of the computing device 200, from the application data 266, or from the media server 101 via the I / O interface 239. In various embodiments, the media item may be an image, a video, a series of images (e.g., GIF), etc.

[0035] The object detection module 202 detects one or more distracting objects within a media item. In some embodiments, the object detection module 202 classifies the distracting objects into categories including power lines, utility poles, or towers. In some embodiments, the object detection module 202 identifies the distracting objects using an object priority, where the object priority is a category of objects having specified parameters. For example, the object priority may include a set of images identified as corresponding to power lines, utility poles, towers, etc. In some embodiments, the object detection module 202 uses an object detection machine learning model that is trained to receive a media item as input and output an identification of a distracting object within the media item.

[0036] In some embodiments, the segmentation module 204 includes (and optionally performs the training of) a trained model, referred to herein as a segmentation machine learning model. In some embodiments, the segmentation module 204 is configured to apply the machine learning model to input data, such as application data 266 (e.g., media items captured by the user device 115), and output a segmentation mask. In some embodiments, the segmentation module 204 may include code executed by the processor 235. In some embodiments, the segmentation module 204 may be stored in memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0037] In some embodiments, the segmentation module 204 may specify circuitry (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 235 to apply the segmentation machine learning model. In some embodiments, the segmentation module 204 may include software instructions, hardware instructions, or a combination. In some embodiments, the segmentation module 204 may provide an application programming interface (API) that may be used by the operating system 262 and / or other applications 264 to invoke the segmentation module 204, e.g., to apply the machine learning model to application data 266 and output a segmentation mask.

[0038] In some embodiments, the segmentation module 204 uses training data to train the segmentation machine learning model. The training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission is provided for use as training data for machine learning, etc. In some embodiments, the training may occur on the media server 101, which provides the training data directly to the user device 115, the training may occur locally on the user device 115, or it may be a combination of both. As described in more detail below, in some embodiments, the training occurs on the media server 101, and a simplified version of the trained segmentation machine learning model is used locally on the user device 115.

[0039] In some embodiments, the training data includes a first set of media items and a second set of media items. The first set of media items is similar to the second set of media items, except that the first set of media items includes distracting objects that are manually subdivided. For example, the distracting objects may be manually annotated and subdivided by a user. In some embodiments, the training data classifies the distracting objects into three separate categories: power lines, poles, and towers. In some embodiments, the training data may include additional categories, such as traffic lights.

[0040] In some embodiments, the training data may include synthetic data generated for training purposes, such as data that is not based on activity in the training context, e.g., data generated from simulated or computer-generated images / videos. The training data may include synthetic media items with distracting objects. In some embodiments, the synthetic media includes distracting objects added in front of outdoor environmental objects, such as power lines, utility poles, and / or towers that are added to images taken in an outdoor environment.

[0041] Referring to FIG. 3, an example 300 for generating synthetic images for training a segmentation machine learning model is shown. The images in the first column are original images, the images in the second column are segmentation masks, and the images in the third column show the corresponding synthesis results. In this example, the synthetic images include synthetic image patches that are synthesized to look realistic. For example, the synthetic image patches include an image with distracting objects, such as power lines, that occlude the subject of the image and make up 1%-30% of the media items, and distracting objects in the background where more than 60% of the sky is included in the image, and synthesizing foreground occluders against the background. Each synthetic image patch includes the synthesis result, a segmentation mask of the occluders (i.e., the distracting objects that occlude the background), and the ground truth background.

[0042] In some embodiments, the training data is augmented if the segmentation mask (e.g., on a power line) breaks down or is weak, the color changes in certain intermediate portions (e.g., due to lighting or other reasons). In another example, the segmentation mask may fail to identify a sign attached to a pole. The segmentation module 204 augments the training data to improve the quality of training of the segmentation machine learning model by making distracting objects more difficult to detect.

[0043] In some embodiments, if continuity of the power lines is assumed, the segmentation module 204 fits a line model onto the power lines to connect the disrupted portions. In some embodiments where the background is complex, the segmentation module 204 generates a synthetic media item by adding power lines and poles in front of the trees instead of using only the trees for training.

[0044] The segmentation module 204 may augment the training data to make distracting objects less visible. In some embodiments, the training data includes broken power lines, which are more difficult to segment because it may be difficult to detect the difference between a broken power line and a new object. In some embodiments, the segmentation module 204 performs power line augmentation to improve the segmentation of one or more broken power lines in the training data. In other words, the segmentation module 204 compensates for one or more broken power lines in the media item.

[0045] In some embodiments, the segmentation module 204 performs enhancement of power lines (or other distracting objects) by modifying contrast. The contrast may be increased or decreased depending on the attributes of the media item. The contrast may be applied locally to target one or more distracting objects or globally to the entire media item.

[0046] Referring to FIG. 4, an example image 400 is shown that includes different local contrasts. For example, in the first row 405 of the images below, decreasing the local contrast makes it more difficult to identify the power lines for segmentation because they resemble the mountains and clouds in the background. In the second row 410 of the images in FIG. 4, increasing the local contrast makes it more difficult to identify the power lines because they become the same shade of black as part of the KODAK sign. As a result, the segmentation module 204 modifies the contrast by decreasing or increasing the contrast depending on the media item to increase the difficulty, thereby improving the training of the segmentation machine learning model.

[0047] In some embodiments, the segmentation module 204 blends the augmented image with the original media item using a checkerboard mask to increase the randomness of the augmentation. In some embodiments, the checkerboard mask W=l(x≠y) is generated using the following formula:

[0048]

number

[0049] where f cb is the checkerboard function, I is the image, and I a is the enhanced image.

[0050] Figure 5 shows an example of an original image, an enhanced image, and a blended image 500. As described with reference to Figure 4, the enhanced image is generated by reducing local contrast. The blended image is therefore a random blend of the original image and an enhanced version with reduced local contrast.

[0051] In some embodiments, the segmentation module 204 modifies the contrast by applying a gain curve to the image, where the gain curve is defined by adding two bias curves. The bias curve is defined by the following equation:

[0052]

number

[0053] where f b is the bias curve function, and b is an adjustable parameter in (0,1). If b<1 / 2, the function is concave, otherwise it is convex.

[0054] The following gain curve equation adds the two bias curves:

[0055]

number

[0056] where f g is a function of the gain curve. When b<1 / 2, the function is sigmoidal, and when the segmentation module 204 applies b less than 0.5 to an image, the function increases contrast. Otherwise, the b value results in a decrease in contrast. FIG. 6 is an example 600 of the gain curve function and how the value of b affects the contrast of an image. When b=0.25, the contrast increases. When b=0.75, the contrast decreases.

[0057] To change the contrast only around distracting objects (e.g., power lines), the segmentation module 204 may make b a function of the mask value m. When m=0, the segmentation module 204 scales the gain function back by 1 / 2 so that it has no effect. The segmentation module 204 still needs b0, which represents the b value at full intensity, i.e., m=1:

[0058]

number

[0059] FIG. 7 shows an example 700 of an original image and an image with increased local contrast generated using equation (4) above.

[0060] In some embodiments, the segmentation module 204 increases the local contrast when it should actually decrease the contrast. This is because the definition of contrast is not the contrast in a local region, but the contrast of a local region relative to the entire image. The goal of data augmentation is to make the power lines less distinguishable from the surrounding sky pixels, in which case the segmentation module 204 decreases the contrast in the local region. The segmentation module 204 adjusts the local contrast relative to the global image to get better results. FIG. 8 shows an example 800 of contrast between different objects as a function of the bias curve.

[0061] In some embodiments, the segmentation module 204 simulates brightness changes by applying a gradient change in contrast in the segmentation mask. In some embodiments, the segmentation module 204 applies a global brightness to the media item to hide distracting objects in the background. With reference to FIG. 9, an example 900 includes an image having varying brightness. FIG. 9 includes an original image, a dark image, and a bright image. Power lines in the darkened image are easier to detect. As a result, the segmentation module 204 applies a global brightness that increases the difficulty of detecting the power lines.

[0062] In some embodiments, the segmentation module 204 uses the bias curve to modify the global brightness of the image. If b<1 / 2, the segmentation module 204 darkens the image. Otherwise, the segmentation module 204 brightens the image. Figure 10 shows an example 1000 of how different bias curves affect an image.

[0063] In some embodiments, the segmentation module 204 augments the training data by applying an augmentation mask to the segmentation mask of the distracting objects in the media item, where the augmentation is used to account for imprecision in the augmentation mask and to create a smooth transition between the distracting object pixels and the background pixels.

[0064] Instead of performing binary enhancement, in some embodiments, the segmentation module 204 applies smooth enhancement to allow a transition according to the following formula:

[0065]

number

[0066] where f dis the enhancement function, I is the image, and F is the enhancement filter. The structure of the filter affects the result of the enhancement. The filter is defined by the following equation:

[0067]

number

[0068] where d is the filter size and p is an adjustable parameter that takes a value in (0,1).

[0069] When p is small, the segmentation module 204 produces a large but weak halo. When p is large, the segmentation module 204 produces a strong enhancement in the center. Figure 11 shows an example 1100 of a segmentation mask with a particular p value. When p = 0.5, the halo has a weak enhancement. When p = 2.0, the halo has a strong enhancement and the lines of the segmentation mask are less distinct than the lines of the segmentation mask when p = 0.5.

[0070] In some embodiments, the blurring can blend into neighboring pixels and make the power lines less visible, so the segmentation module 204 blurs around the power lines to avoid easier cases for false positives, such as other cables or fences. The local blurring operation may be defined as follows:

[0071]

number

[0072] where f lb is a function of local blur, M is the enhanced power line mask, and g is a blur function (e.g., Gaussian). Figure 12 shows an example 1200 of an original image and an image with local blur and enhanced segmentation mask.

[0073] In some embodiments, the segmentation module 204 adds negative examples to the training. In some embodiments, the training data includes 1 / 3 negative examples and the classes include trees, fences, and buildings. The segmentation module 204 may increase the amount and diversity of negative examples. The additional classes may include wire, wire rope, barbed wire, wire fencing, picket fence, and fence post.

[0074] To avoid the situation where the segmentation module 204 generates a segmentation mask for a utility pole over a vertical structure that is not a utility pole, the segmentation module 204 applies heuristics during inference time. For example, this can cause the segmentation module 204 to eliminate utility poles without attaching power lines. In some embodiments, the segmentation module 204 performs a breadth-first search (BFS) to get the connected components. Then, for each utility pole component, the segmentation module 204 calculates the shortest distance to the power line. If the power line is too far away (which can be an adjustable parameter), the segmentation module 204 removes the entire utility pole object. Assuming there are N pixels and K components, the BFS is O(N). The shortest distance is the bottleneck when the segmentation module 204 iterates over pixel pairs, and the complexity is O(N 2 ). The segmentation module 204 can simplify the object as a rotated box, which reduces the complexity to O(K 2 ) to reduce the

[0075] In some embodiments, the segmentation module 204 stored in the media server 101 is trained on the training data to generate a high-capacity model. The segmentation module 204 then generates a low-capacity model, including a more compact neural network model, by performing inference on a much larger dataset, which is subdivided by the high-capacity model. In that way, the high-capacity machine learning model is refined into a smaller, i.e., less resource-intensive, trained segmentation machine learning model. The trained segmentation machine learning model is used by the segmentation module 204 stored in the user device 115. Since local operations such as blur and contrast are computationally expensive, in some embodiments, the segmentation module 204 pre-computes local operations and adds them to the dataset. Global operations (brightness) are computationally cheap, and therefore the segmentation module 204 performs its analysis on the fly during training and adds randomness to gain diversity.

[0076] The trained machine learning model may include one or more model formats or structures. For example, the model formats or structures may include any type of neural network, such as a linear network, a deep learning neural network implementing multiple layers (e.g., a "hidden layer" between the input layer and the output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), or a sequence-to-sequence neural network (e.g., a network that receives as input a sequence of data, such as words in a sentence, frames in a video, etc., and produces as output an array of results).

[0077] The model format or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, the nodes of a first layer (e.g., an input layer) may receive data as input data or application data. Such data may include, for example, one or more pixels per node, for example, when the trained model is used, for example, for analyzing media items. Subsequent intermediate layers may receive as input the output of the nodes of the previous layer according to the connectivity specified in the model format or structure. These layers may also be referred to as hidden layers. For example, the first layer may output a segmentation between the foreground and background. The final layer (e.g., an output layer) generates the output of the machine learning model. For example, the output layer may receive the segmentation of the media item into the foreground and background and output whether the pixel is part of the segmentation mask. In some embodiments, the model format or structure also specifies the number and / or type of nodes in each layer.

[0078] In different embodiments, the trained model may include one or more models. One or more of the models may include multiple nodes arranged in layers per model structure or morphology. In some embodiments, the multiple nodes may be memoryless computational nodes configured to process a unit of input and generate a unit of output. The computation performed by a node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate a node output. In some embodiments, the computation performed by a node may include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computation may include operations such as matrix multiplication. In some embodiments, the computation by the multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or using dedicated neural circuitry. In some embodiments, a node may include a memory, e.g., to store and use one or more previous inputs in processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node. An LSTM node may use memory to maintain a "state" that allows the node to operate like a finite state machine (FSM).

[0079] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may begin as a number of nodes organized into layers as specified by the model format or structure. At initialization, respective weights may be applied to the connections between each pair of nodes that are connected according to the model format, e.g., nodes in successive layers of a neural network. For example, the respective weights may be randomly assigned or initialized to default values. The model may then be trained, e.g., using training data, to generate results.

[0080] Training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., manually annotated segments and synthesized media items) and corresponding ground truth outputs for each input (e.g., ground truth segmentation masks that accurately identify distracting objects in the media item). Based on a comparison of the model's output to the ground truth outputs, the values ​​of the weights are, for example, automatically adjusted to increase the probability that the model generates a ground truth output for the media item.

[0081] In some embodiments, during training, the segmentation module 204 outputs a segmentation mask along with a segmentation score that indicates the quality of the segmentation mask in identifying power lines, poles, and / or towers in the media item. The segmentation score may reflect the intersection of the union (loU) between the segmentation mask output by the segmentation machine learning model and the ground truth segmentation mask. In some embodiments, the segmentation score is a number relative to a total number, such as 40 / 100. Other representations of the segmentation score are possible.

[0082] In various embodiments, the trained model includes a set of weights or embeddings that correspond to the model structure. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights. In various embodiments, the trained model includes a set of weights or embeddings that correspond to the model structure. In embodiments where data is omitted, the segmentation module 204 may include a trained model that is based on previous training, e.g., by a developer of the segmentation module 204, by a third party, etc. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0083] In some embodiments, the segmentation module 204 receives a media item. The segmentation module 204 provides the media item as an input to a trained machine learning model. In some embodiments, the trained machine learning model outputs a segmentation mask for one or more distracting objects within a crop of the media item and a corresponding segmentation score that indicates the quality of the segmentation mask.

[0084] In some embodiments, the machine learning model outputs a confidence value for each segmentation mask output by the trained machine learning model. The confidence value may be expressed as a percentage, a number between 0 and 1, etc. For example, the machine learning model outputs a confidence value of 85% for confidence that the segmentation mask correctly covered one or more distracting objects.

[0085] The restoration module 206 generates an output media item that is free of (eliminated from) the one or more distracting objects. In some embodiments, the restoration module 206 includes a set of instructions executable by the processor 235 to generate the output media item. In some embodiments, the restoration module 206 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0086] In some embodiments, the inpainting module 206 receives a segmentation mask from the segmentation module 204. The inpainting module 206 performs inpainting of portions of a media item that match the segmentation mask. For example, the inpainting module 206 replaces all pixels in the segmentation mask with pixels that match the background in the media item. In some embodiments, the pixels that match the background may be based on another media item in the same location. In some embodiments, the inpainting module 206 replaces pixels identified by the segmentation mask with pixels that match adjacent regions in the media item.

[0087] In some embodiments, the restoration module 206 receives a media item and a segmentation mask as input from the segmentation module 204 and trains a restoration machine learning model to output an output media item that is free of one or more distracting objects in the output media item.

[0088] In some embodiments, the repair module 206 generates a repair reliability metric that is used to determine the quality of the output media item. In some embodiments, an evaluator (e.g., an evaluation machine learning model) is used to determine the quality of the output media item. The quality may be based on whether the distracting object was correctly removed (e.g., the distracting object and all of its associated features, such as shadows, were removed) and whether the repair is not discernible (e.g., the repaired area of ​​the output media item blends with the rest of the media item such that the removal of the distracting object does not result in a perceptible inaccuracy).

[0089] The user interface module 208 generates a user interface. In some embodiments, the user interface module 208 includes a set of instructions executable by the processor 235 to generate a user interface. In some embodiments, the user interface module 208 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0090] The user interface module 208 generates a user interface that asks the user for permission to access the user's media items before performing any of the steps performed by the modules of FIG. 2 and described with reference to FIG. 14.

[0091] The user interface module 208 generates a user interface that includes the media item and provides suggestions for removing one or more distracting objects. For example, the user interface may include an outline surrounding each distracting object. In some embodiments, the user interface module 208 generates a suggestion to automatically remove the distracting object if the distracting object is detected with a threshold level of confidence and if the quality of the output media item meets a quality threshold. This can ensure that the user experience of automatic removal of distracting objects is satisfactory.

[0092] The user interface module 208 generates a user interface that includes the repaired output media item. In some embodiments, the output media item may be labeled (visually) or marked (with a code, e.g., steganographically) to indicate that the output media item has been edited to erase one or more objects. In some embodiments, the user interface includes options for editing the output media item, sharing the output media item, adding the output media item to a photo album, etc. Options for editing the output media item may include the ability to undo the erasure of objects.

[0093] In some embodiments, the user interface module 208 receives feedback from users on the user devices 115. The feedback may take the form of users posting output media items, users deleting output media items, users sharing output media items, etc.

[0094] Exemplary Methods 13 shows a flowchart of an example method 1300 for training a segmentation machine learning model. The method 1300 of FIG. 13 may begin at block 1302. The method 1300 shown in the flowchart may be performed by the computing device 200 of FIG. 2. In some embodiments, the method 1300 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0095] At block 1302, training data is generated that includes a first set of media items and a second set of media items, the first set of media items corresponding to the second set of media items and including distracting objects that are to be manually segmented. Block 1302 may be followed by block 1304.

[0096] At block 1304, a segmentation machine learning model is trained based on the training data to receive a media item having one or more distracting objects and output a segmentation mask for one or more segmented objects corresponding to the one or more distracting objects.

[0097] 14 shows a flowchart of an example method 1400 for removing distracting objects from a media item. The method 1400 of FIG. 14 may begin at block 1402. The method 1400 shown in the flowchart may be performed by the computing device 200 of FIG. 2. In some embodiments, the method 1400 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0098] At block 1402, user permission to perform method 1400 is received. For example, a user may load an application to view a media item, but before the media item is displayed, a user interface asks for user permission to access the media item associated with the user. The user interface may also ask for permission to suggest the media item, to modify the media item, to allow the user to allow access to only certain media items, to ensure that media items are not stored or transferred to a server without user permission, etc. Block 1402 may be followed by block 1404.

[0099] At block 1404, it is determined whether user permission has been received. If user permission has not been received, block 1404 is followed by block 1406, which stops the method 1400. If user permission has been received, block 1404 is followed by block 1408.

[0100] At block 1408, a media item is received from a user. For example, the media item may include a distracting object such as a power line, a utility pole, or a tower. Block 1408 may be followed by block 1410.

[0101] One or more distracting objects are identified in a media item at block 1410. Block 1410 may be followed by block 1412.

[0102] The media items are provided to the trained segmentation machine learning model at block 1412. Block 1412 may be followed by block 1414.

[0103] At block 1414, a segmentation mask for one or more distracting objects in the media item is output along with the trained segmentation machine learning model. Block 1414 may be followed by block 1416.

[0104] At block 1416, the portion of the media item that matches the segmentation mask is inpainted to obtain an output media item in which the one or more distracting objects are absent from the output media item.

[0105] In addition to the above, a user may be provided with controls that allow the user to select both whether and when the systems, programs, or features described herein may enable collection of user information (e.g., information about the user's media items, including images and / or videos, social networks, social actions or activities, occupation, the user's preferences (e.g., regarding objects in images), or the user's current location), as well as whether the user is sent content or communications from the server. Additionally, certain data may be treated in one or more ways before being stored or used such that personally identifiable information is removed. For example, the user's identity may be treated such that no personally identifiable information can be determined for that user, or location information is obtained (e.g., to the city, ZIP code, or state level) such that the user's geographic location is generalized and the user's specific location cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0106] In the above description, for the purpose of explanation, numerous specific details are set forth to provide a thorough understanding of the present specification. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the embodiments may be described above primarily with reference to user interfaces and specific hardware. However, the embodiments may be applied to any type of computing device capable of receiving data and commands, as well as any peripheral device that provides services.

[0107] Reference herein to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with an embodiment or instance may be included in at least one implementation of the description. The appearances of the phrase "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiments.

[0108] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0109] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, as will be apparent from the discussion below, throughout the description, discussions utilizing terms including "processing" or "calculating" or "computing" or "determining" or "displaying" and the like will be understood to refer to operations and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the computer system's registers and memory, or converts it into other data represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0110] The embodiments herein may also relate to a processor for performing one or more steps of the above-mentioned methods. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including any type of disk including, but not limited to, an optical disk, a ROM, a CD-ROM, a magnetic disk, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory including a USB key having a non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0111] This specification can take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments containing both hardware and software elements, hi some embodiments, this specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0112] Furthermore, the descriptions may take the form of a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any apparatus that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device.

[0113] A data processing system suitable for storing or executing program code includes at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory used during actual execution of the program code, mass storage devices, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from mass storage devices during execution.

Claims

1. 1. A computer-implemented method comprising: generating training data including a first set of media items and a second set of media items, the first set of media items including a distracting object and the second set of media items including a manual subdivision of the distracting object, the method further comprising: identifying one or more original media items in the first set of media items that include one or more broken power lines; generating one or more corrected media items that correct the one or more broken power lines; generating one or more augmented media items for the training data by mixing a portion of the one or more compensated media items with a portion of each of the one or more original media items to increase randomness of the augmentation; 11. A method comprising: receiving a media item having one or more distracting objects; and training a segmentation machine learning model based on the training data to output segmentation masks for one or more segmented objects corresponding to the one or more distracting objects.

2. The method of claim 1, wherein generating the one or more enhanced media items includes blending the one or more corrected media items with the respective one or more original media items using a checkerboard mask.

3. Generating the one or more corrected media items to correct the one or more broken power lines includes: The method of claim 1 , comprising modifying local contrast in the one or more original media items to generate one or more corresponding enhanced media items.

4. The method of claim 3 , wherein the local contrast is modified using a gain curve that adds two bias curves.

5. 2. The method of claim 1, wherein generating the training data comprises augmenting one or more of the first set of media items by applying augmentation to a segmentation mask of one or more distracting objects.

6. The method of claim 1 , wherein the one or more distracting objects are organized into categories, the categories including at least one selected from the group of: power lines, utility poles, towers, and combinations thereof.

7. Training the segmentation machine learning model includes: generating a first machine learning model based on the training data; and refining the first machine learning model into a trained segmentation machine learning model by performing inference on training data segmented by the first machine learning model.

8. The method of claim 1 , wherein the training data further comprises synthetic images in which the distracting object is added in front of an outdoor environment object.

9. 1. A computer-implemented method for removing distracting objects from a media item, comprising: Receiving a media item from a user; identifying one or more distracting objects within the media item; providing the media items to a trained segmentation machine learning model; using the trained segmentation machine learning model to output a segmentation mask for one or more distracting objects in the media item; and inpainting portions of the media item that match the segmentation mask to obtain an output media item, wherein the one or more distracting objects are absent from the output media item; The trained segmentation machine learning model comprises: identifying one or more original media items in a first set of media items that include one or more broken power lines; generating one or more corrected media items that correct the one or more plurality of broken power lines; and The method is trained by generating training data by generating one or more augmented media items for the training data by mixing portions of the one or more corrected media items with portions of respective one or more original media items to increase the randomness of the augmentation.

10. The method of claim 9 , wherein the one or more distracting objects are organized into categories, the categories including at least one selected from the group of: power lines, utility poles, towers, and combinations thereof.

11. The method of claim 9 , further comprising providing a user with suggestions to remove the one or more distracting objects from a media item.

12. 10. The method of claim 9, wherein the trained segmentation machine learning model is trained using training data including the first set of media items and a second set of media items, the first set of media items including a distracting object, and the second set of media items including a hand fragmentation of the distracting object.

13. A program for causing one or more computers to execute the method according to any one of claims 1 to 12.

Images