Segmenting and removing objects from media items

Abstract

A media application generates training data including a first set of media items and a second set of media items, where the first set of media items correspond to the second set of media items and include manually segmented distractor objects. The media application trains a segmentation machine learning model based on the training data to receive a media item having one or more distractor objects and output a segmentation mask of one or more segmented objects corresponding to the one or more distractor objects.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 257,114, filed October 18, 2021, entitled “Segmenting and Removing Objects from Media Items,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] The user-perceived quality of visual media items—such as images (still images, images with selective motion, etc.) and videos—can be improved by removing certain objects that distract from the focus of the media item. Distracting objects can be removed manually, but this task can be arduous and incomplete. Furthermore, it is difficult to remove distracting objects automatically from media items because such removal may result in the removal of additional objects in the image or parts of an object (misidentified as distracting objects); over-triggered and unrealistic results where extra objects are removed; or incomplete segmentation where parts of the removed object remain visible.

[0004] Users can manually remove distracting objects using image or video editing techniques. However, this task can be arduous and incomplete. Furthermore, automatically removing distracting objects is difficult because it may result in false alarms, such as the removal of other objects or parts of objects, or incomplete segmentation leading to false alarms where parts of removed objects remain visible.

[0005] The background description provided herein is intended to generally present the context of this disclosure. The work of the currently named inventors described in this background section, and descriptions that may not conform to the prior art at the time of filing, are neither expressly nor implied to be acknowledged as prior art to this disclosure. Summary of the Invention

[0006] The computer-implemented method includes generating training data comprising a first set of media items and a second set of media items, wherein the first set of media items corresponds to the second set of media items and includes manually segmented perturbation objects. The method also includes training a segmentation machine learning model based on the training data to receive media items having one or more perturbation objects and output a segmentation mask corresponding to one or more segmented objects of the one or more perturbation objects.

[0007] In some embodiments, one or more interfering objects are power lines, and the method further includes identifying one or more media items from a first set of media items including one or more disconnected power lines, and enhancing one or more media items to correct one or more disconnected power lines in training data. In some embodiments, enhancing one or more media items to correct one or more disconnected power lines includes: modifying local contrast in one or more media items to generate corresponding one or more enhanced media items, and blending a portion of one or more media items with a portion of corresponding one or more enhanced media items. In some embodiments, a gain curve that sums two bias curves is used to modify local contrast. In some embodiments, generating training data includes enhancing one or more media items by applying expansion to a segmented mask of one or more interfering objects. In some embodiments, one or more interfering objects are organized into categories including at least one group selected from power lines, utility poles, towers, and combinations thereof. In some embodiments, training a segmented machine learning model includes: generating a high-capacity machine learning model based on training data, and refining the high-capacity machine learning model into a trained segmented machine learning model by running inference about the training data segmented by the high-capacity machine learning model. In some embodiments, the training data also includes a synthetic image in which the interfering objects are added in front of an outdoor environment object.

[0008] In some embodiments, a computer-implemented method for removing interfering objects from a media item includes: receiving a media item from a user, identifying one or more interfering objects in the media item, providing the media item to a trained segmented machine learning model, using the trained segmented machine learning model, outputting a segmented mask of one or more interfering objects in the media item, and repairing a portion of the media item that matches the segmented mask to obtain an output media item, wherein one or more interfering objects are not present in the output media item.

[0009] In some embodiments, one or more interfering objects are organized into categories, including at least one group selected from power lines, utility poles, towers, and combinations thereof. In some embodiments, the method further includes providing a user with suggestions to remove one or more interfering objects from media items. In some embodiments, a trained segmented machine learning model is trained using training data including a first group of media items and a second group of media items, wherein the first group of media items corresponds to the second group of media items, and includes manually segmented interfering objects.

[0010] In some embodiments, a non-transitory computer-readable medium has instructions stored thereon that, when executed by one or more computers, cause one or more computers to perform operations including: generating training data including a first set of media items and a second set of media items, wherein the first set of media items corresponds to the second set of media items and includes manually segmented perturbation objects; and training a segmented machine learning model based on the training data to receive media items having one or more perturbation objects and output a segmented mask corresponding to one or more segmented objects of the one or more perturbation objects.

[0011] In some embodiments, the one or more interfering objects are power lines, and the operation further includes: identifying one or more media items from a first set of media items including one or more disconnected power lines, and enhancing one or more media items to correct one or more disconnected power lines in the training data. In some embodiments, enhancing one or more media items to correct one or more disconnected power lines includes: modifying the local contrast in one or more media items to generate a corresponding one or more enhanced media items, and blending a portion of one or more media items with a portion of the corresponding one or more enhanced media items. In some embodiments, a gain curve that sums two bias curves is used to modify the local contrast. In some embodiments, generating training data includes enhancing one or more media items by applying an expansion to a segmented mask of one or more interfering objects. In some embodiments, the one or more interfering objects are organized into categories, including at least one group selected from power lines, utility poles, towers, and combinations thereof. In some embodiments, training a segmented machine learning model includes: generating a high-capacity machine learning model based on the training data, and refining the high-capacity machine learning model into a trained segmented machine learning model by running inference about the training data segmented by the high-capacity machine learning model. In some embodiments, the training data also includes a synthetic image in which the interfering objects are added in front of an outdoor environment object.

[0012] The techniques described in the specification advantageously describe a media application that includes a trained segmented machine learning model that overcomes the technical difficulties of identifying distracting objects in media items. Attached Figure Description

[0013] The application documents contain at least one drawing executed in color. The Patent Office will, upon request, provide a copy of the patent or patent application publication with color drawings and pay the necessary fees.

[0014] Figure 1 This is a block diagram of an example network environment for removing interfering objects from media items according to some embodiments described herein.

[0015] Figure 2This is a block diagram of an example computing device for removing interfering objects from media items according to some embodiments described herein.

[0016] Figure 3 Examples of generating synthetic images for training segmented machine learning models are shown, according to some embodiments described herein.

[0017] Figure 4 Example images with different local contrasts are shown according to some embodiments described herein.

[0018] Figure 5 Example original images, enhanced images, and blended images based on some embodiments described herein.

[0019] Figure 6 Examples are shown illustrating bias curves according to some embodiments described herein and how they affect image contrast.

[0020] Figure 7 Example original images and example images with increased local contrast are shown according to some embodiments described herein.

[0021] Figure 8 Examples of contrast between different objects based on bias curves are shown according to some embodiments described herein.

[0022] Figure 9 Example images with different brightness levels are shown according to some embodiments described herein.

[0023] Figure 10 Examples of images affected by different bias curves according to some embodiments described herein are shown.

[0024] Figure 11 Variations of segmented masks based on different expansions are shown according to some embodiments described herein.

[0025] Figure 12 Example original images and example images with local blur and expanded segmented masks are shown according to some embodiments described herein.

[0026] Figure 13 A flowchart is shown illustrating an example method for training a segmented machine learning model according to some embodiments described herein.

[0027] Figure 14 A flowchart illustrating an example method for removing interfering objects from a media item according to some embodiments described herein is shown. Detailed Implementation

[0028] Example Environment 100

[0029] Figure 1 A block diagram of an example environment 100 is shown. In some embodiments, environment 100 includes a media server 101, user device 115a, and user device 115n coupled to network 105. Users 125a and 125n may be associated with corresponding user devices 115a and 115n. In some embodiments, environment 100 may include Figure 1 Other servers or devices not shown. Figure 1 In the other figures, letters following a reference number (e.g., "115a") indicate a reference to the element having that specific reference number. Reference numbers without subsequent letters in the text, such as "115", indicate a general reference to an embodiment of the element with that reference number.

[0030] Media server 101 may include a processor, memory, and network communication hardware. In some embodiments, media server 101 is a hardware server. Media server 101 is communicatively coupled to network 105 via signal line 102. Signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as... Or other wireless technologies. In some embodiments, media server 101 transmits data to one or more of user devices 115a, 115n and receives data from one or more of user devices 115a, 115n via network 105. Media server 101 may include media application 103a and database 199.

[0031] Database 199 can store machine learning models, training datasets, images, etc. With the user's consent, Database 199 can also store social network data associated with user 125, user preferences, etc.

[0032] User equipment 115 may be a computing device including memory coupled to a hardware processor. For example, user equipment 115 may include a mobile device, tablet computer, mobile phone, wearable device, head-mounted display, mobile email device, portable game player, portable music player, e-reader device, or another electronic device capable of accessing network 105.

[0033] In the illustrated implementation, user equipment 115a is coupled to network 105 via signal line 108, and user equipment 115n is coupled to network 105 via signal line 110. Media application 103 can be stored on user equipment 115a as media application 103b and / or on user equipment 115n as media application 103c. Signal lines 108 and 110 can be wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections, such as… Or other wireless technologies. User equipment 115a and 115n are accessed by users 125a and 125n, respectively. Figure 1 User equipment 115a and 115n are used as examples. Although Figure 1 Two user equipments 115a and 115n are shown, but the disclosure applies to system architectures with one or more user equipments 115.

[0034] Media application 103 may be stored on media server 101 and / or user device 115. In some embodiments, the operations described herein are performed on media server 101 or user device 115. In some embodiments, some operations may be performed on media server 101 and some on user device 115. The execution of operations conforms to user settings. For example, user 125a may specify that an operation will be performed on the corresponding device 115a instead of on media server 101. With such a setting, the operations described herein are performed entirely on user device 115a, without any operation performed on media server 101. Furthermore, user 125a may specify that the user's images and / or other data are stored only on the local user device 115a, and not on media server 101. With such a setting, no user data is transmitted or stored on media server 101. User data is transmitted to media server 101, temporarily or permanently stored by media server 101, and operations performed by media server 101 on such data are performed only if the user has consented to the transmission, storage, and execution of operations by media server 101. Users are given the option to change settings at any time, for example, enabling or disabling the use of media server 101.

[0035] Machine learning models (e.g., neural networks or other types of models), if used for one or more operations, are stored locally and utilized locally on user device 115 with specific user permission. Server-side models are used only with user permission. Model training is performed using synthetic datasets, as referenced below. Figure 5 Furthermore, a trained model can be provided for use on user device 115. During such use, model training can be performed on the device if permitted by user 125. Updated model parameters can be transmitted to media server 101, for example, with joint learning enabled, if permitted by user 125. The model parameters do not include any user data.

[0036] Media application 103 trains a segmented machine learning model to remove one or more distracting objects from media items by generating training data that includes a first set of media items and a second set of media items, where the first set of media items corresponds to the second set of media items and includes manually segmented distracting objects. In this paper, distracting objects are objects belonging to a predetermined object category that are often unwanted in an image, and the technique described herein can seamlessly remove them from an image by replacing the "distracting objects" with the same background. Examples of distracting objects used for illustration here are power lines, poles, or towers. Media application 103 can train a segmented machine learning model based on training data to receive media items with one or more distracting objects and output a segmented mask corresponding to one or more segmented objects of the one or more distracting objects.

[0037] In some embodiments, media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a may be implemented using a combination of hardware and software.

[0038] Example computing device 200

[0039] Figure 2 This is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing media application 103a. In another example, the computing device 200 is a user device 115.

[0040] In some embodiments, the computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all of which are coupled via a bus 218. The processor 235 may be coupled to the bus 218 via signal line 222, the memory 237 may be coupled to the bus 218 via signal line 224, the I / O interface 239 may be coupled to the bus 218 via signal line 226, the display 241 may be coupled to the bus 218 via signal line 228, the camera 243 may be coupled to the bus 218 via signal line 230, and the storage device 245 may be coupled to the bus 218 via signal line 232.

[0041] Processor 235 may be one or more processors and / or processing circuitry to execute program code and control the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a system comprising a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing functionality, a dedicated processor for implementing neural network-based processing, neural circuits, processors optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors implementing neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce probabilistic outputs; for example, the output produced by processor 235 may be imprecise or accurate within the expected output range. Processing is not necessarily limited to a specific geographical location or has time constraints. For example, a processor may perform its functions in real-time, offline, batch processing modes, etc. A portion of the processing can be executed at different times and in different locations by different (or the same) processing systems. The computer can be any processor that communicates with memory.

[0042] Memory 237 is disposed in computing device 200 and accessible to processor 235. It can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions executed by the processor or processor group, and positioned separately from and / or integrated with processor 235. Memory 237 may store software executed by processor 235 on computing device 200, including media application 103.

[0043] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, image library applications, image management applications, picture gallery applications, communication applications, web hosting engines or applications, media sharing applications, etc. One or more methods disclosed herein can run in various environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0044] Application data 266 may be data generated by other applications 264 or the hardware of computing device 200. For example, application data 266 may include images used by an image gallery application and user actions identified by other applications 264 (e.g., a social networking application).

[0045] I / O interface 239 can provide functionality that enables computing device 200 to interface with other systems and devices. Interface devices may be included as part of computing device 200 or may be separate from and communicate with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0046] Examples of interface devices capable of connecting to I / O interface 239 may include display 241, which can be used to display content, such as images, videos, and / or user interfaces of output applications as described herein, and to receive touch (or gesture) input from a user. For example, display 241 may be used to display a user interface including graphical guidance on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), a light-emitting diode (LED) or plasma display, a cathode ray tube (CRT), a television, a monitor, a touch screen, a 3D display, or other visual display device. For example, display 241 may be a flat panel display on a mobile device, multiple displays embedded in an eyeglass-shaped or headphone device, or a monitor screen for a computer device.

[0047] Camera 243 can be any type of image capture device capable of capturing media items, including images and / or videos. In some embodiments, camera 243 captures images or videos provided by I / O interface 239 to media application 103.

[0048] Storage device 245 stores data related to media application 103. For example, storage device 245 may store training datasets, which include labeled images, machine learning models, outputs from machine learning models, etc.

[0049] Figure 2 An example of a media application 103 stored in memory 237 is shown, which includes an object detection module 202, a segmentation module 204, a repair module 206, and a user interface module 208.

[0050] The object detection module 202 identifies interfering objects in a media item. In some embodiments, the object detection module 202 includes a set of instructions executable by the processor 235 to identify interfering objects in a media item. In some embodiments, the object detection module 202 is stored in the memory 237 of the computing device 200 and is accessible and executable by the processor 235.

[0051] In some embodiments, the object detection module 202 receives media items. Media items may be received from the camera 243 of the computing device 200, from application data 266, or from the media server 101 via I / O interface 239. In various embodiments, media items may be images, videos, a series of images (e.g., GIFs), etc.

[0052] Object detection module 202 detects one or more interfering objects in a media item. In some embodiments, object detection module 202 classifies interfering objects into categories including power lines, utility poles, or towers. In some embodiments, object detection module 202 uses object priors to identify interfering objects, where the object prior is an object category with specified parameters. For example, the object prior may include a set of images identified as corresponding to power lines, utility poles, towers, etc. In some embodiments, object detection module 202 uses an object detection machine learning model trained to receive media items as input and output the identification of interfering objects in the media items.

[0053] In some embodiments, segmentation module 204 includes (and optionally also performs training on) a trained model, referred to herein as a segmented machine learning model. In some embodiments, segmentation module 204 is configured to apply the machine learning model to input data, such as application data 266 (e.g., media items captured by user device 115), and output a segmented mask. In some embodiments, segmentation module 204 may include code to be executed by processor 235. In some embodiments, segmentation module 204 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0054] In some embodiments, segmentation module 204 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) enabling processor 235 to apply a segmented machine learning model. In some embodiments, segmentation module 204 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, segmentation module 204 may provide an application programming interface (API) that can be invoked by operating system 262 and / or other applications 264, for example, to apply a segmented machine learning model to application data 266 to output a segmented mask.

[0055] In some embodiments, the segmentation module 204 uses training data to train a segmented machine learning model. The training data can be obtained from any source, such as a data repository specifically tagged for training, providing data licensed for use as machine learning training data, etc. In some embodiments, training may occur on media server 101, which provides training data directly to user device 115, training may occur locally on user device 115, or a combination of both. As discussed in more detail below, in some embodiments, training occurs on media server 101 and a simplified version of the trained segmented machine learning model is used locally on user device 115.

[0056] In some embodiments, the training data includes a first set of media items and a second set of media items. The first set of media items is similar to the second set, except that it includes manually segmented interfering objects. For example, the interfering objects can be manually annotated and segmented by the user. In some embodiments, the training data categorizes the interfering objects into three distinct categories: power lines, utility poles, and towers. In some embodiments, the training data may include additional categories, such as traffic lights.

[0057] In some embodiments, training data may include synthetic data generated for training purposes, such as data not based on activities in the training environment, for example, data generated from simulated or computer-generated images / videos. Training data may include synthetic media items with interfering objects. In some embodiments, the synthetic media includes interfering objects added in front of outdoor environmental objects, such as power lines, utility poles, and / or towers added to images taken in an outdoor environment.

[0058] Go to Figure 3 Example 300 is shown for generating synthetic images used to train a segmented machine learning model. The images in the first column are the original images, the images in the second column are segmented masks, and the images in the third column show the corresponding synthetic results. In this example, the synthetic images include synthetic image tiles that appear realistic after being synthesized. For example, the synthetic image tiles include interfering objects, such as power lines, that obscure the subject of the image and occupy 1%-30% of the media items; an image with interfering objects in the background, where the image includes 60% or more of the sky; and a foreground occluder synthesized on the background. Each synthetic image tile includes the synthetic result, a segmented mask of the occluder (i.e., the interfering objects obscuring the background), and a ground-based background.

[0059] In some embodiments, training data is augmented when the segmented mask is broken (e.g., on a broken power line) or when the color of some intermediate portions becomes blurred (e.g., due to lighting or other reasons). In another example, the segmented mask may fail to recognize a sign attached to a pole. Segmentation module 204 augments the training data to improve the training quality of the segmented machine learning model by making interfering objects more difficult to detect.

[0060] In some embodiments, assuming the continuity of the power lines, the segmentation module 204 fits a line model about the power lines and connects broken sections. In some complex embodiments, the segmentation module 204 generates synthetic media items by adding power lines and poles before the tree, rather than using only the tree used for training.

[0061] Segmentation module 204 can enhance training data to make interfering objects less noticeable. In some embodiments, the training data includes broken power lines, which are more difficult to segment because it is harder to detect the difference between broken power lines and new objects. In some embodiments, segmentation module 204 performs power line enhancement to improve the segmentation of one or more broken power lines in the training data. In other words, segmentation module 204 corrects one or more broken power lines in a media item.

[0062] In some embodiments, segmentation module 204 performs enhancement of power lines (or other interfering objects) by changing the contrast. The contrast may be increased or decreased, depending on the properties of the media item. The contrast may be applied locally to target one or more interfering objects, or it may be applied globally to the entire media item.

[0063] Go to Figure 4 Example image 400 is shown, which includes varying local contrast. For instance, in the first row 405 of the image below, reduced local contrast makes it more difficult to identify the power lines to be segmented, as they resemble the mountains and clouds in the background. Figure 4 In the second row 410 of the image, increasing the local contrast makes it more difficult to identify the power lines because they become the same black shadows as part of the KODAK logo. As a result, the segmentation module 204 modifies the contrast by decreasing or increasing the contrast depending on the media item, to increase the difficulty and thereby improve the training of the segmented machine learning model.

[0064] In some embodiments, the segmentation module 204 blends the enhanced image with the original media item having a checkerboard mask to increase the randomness of the enhancement. In some embodiments, the checkerboard mask W = 1 (x ≠ y) is generated using the following equation:

[0065] f cb (I, I) a)=[1(x≠y)I]+[1(x=y)I a (Equation 1)

[0066] Where f cb It's a chessboard function, I is an image, and I... a It's an image enhancement.

[0067] Figure 5 Example 500 of original images, enhanced images, and blended images are shown. (See reference...) Figure 4 The enhanced image is generated by reducing local contrast. Therefore, the blended image is a random mixture of the original image and the enhanced version with reduced local contrast.

[0068] In some embodiments, the segmentation module 204 modifies the contrast by applying a gain curve to the image, wherein the gain curve is defined by adding two bias curves. The bias curve is defined by the following equation:

[0069] f b (x; b) = x / [(1 / b-2)(1-x)+1] (Equation 2)

[0070] Where fb is the bias curve function and b is an adjustable parameter falling in (0, 1). The function is concave when b < 1 / 2, and convex otherwise.

[0071] The following gain curve equation adds the two bias curves together:

[0072]

[0073] Where f b It is a function of the gain curve. When b < 1 / 2, the function is S-shaped, and when the segmentation module 204 applies a b value less than 0.5 to the image, the function increases contrast. Otherwise, the b value will cause a decrease in contrast. Figure 6 This is an example of a gain curve function, 600, and how the value of b affects image contrast. When b = 0.25, contrast increases; when b = 0.75, contrast decreases.

[0074] To change the contrast only around the interfering object (e.g., power lines), segmentation module 204 can make b a function of the mask value m. When m = 0, segmentation module 204 makes the function return to 1 / 2, so the gain function has no effect. Segmentation module 204 still needs b0, which indicates the value of b at full intensity, i.e., when m = 1:

[0075] b(m; b0) = 1 / 2 + (b0 - 1 / 2)m (Equation 4)

[0076] Figure 7Example 700 shows the original image and an image with increased local contrast generated using Equation 4 above.

[0077] In some embodiments, segmentation module 204 increases local contrast when contrast should actually be reduced. This is because contrast is defined not as contrast within a local area, but as contrast of a local area relative to the entire image. The purpose of data augmentation is to reduce the distinguishability of power lines from surrounding sky pixels; in this case, segmentation module 204 reduces contrast within the local area. Segmentation module 204 adjusts the local contrast relative to the global image to achieve good results. Figure 8 Example 800 shows the contrast between different objects based on the bias curve.

[0078] In some embodiments, the segmentation module 204 simulates brightness changes by applying a gradient change in contrast within a segmented mask. In some embodiments, the segmentation module 204 applies global brightness to the media item to hide distracting objects in the background. Go to Figure 9 Example 900 includes an image with varying brightness. Figure 9 This includes the original image, a darkened image, and a brightened image. Power lines are easier to detect in the darkened image. As a result, the segmentation module 204 applies global brightness, which increases the difficulty of detecting power lines.

[0079] In some embodiments, the segmentation module 204 uses an offset curve to change the global brightness of the image. When b < 1 / 2, it darkens the image; otherwise, it brightens the image. Figure 10 Example 1000 shows how different bias curves affect an image.

[0080] In some embodiments, the segmentation module 204 augments the training data by applying a dilation mask to a segmented mask of a perturbation object in a media item. The dilation is used to explain inaccuracies in the dilation mask and to create a smooth transition between perturbation object pixels and background pixels.

[0081] In some embodiments, instead of performing binary expansion, segmentation module 204 applies soft expansion to allow a transition with the following equation:

[0082]

[0083] Where f d Let I be the dilation function, I be the image, and F be the dilation filter. The structure of the filter affects the dilation result. The filter is defined using the following equation:

[0084] F(x, y; d, p) = -((2x / d - 1) 2 +(2y / d-1) 2 )p / 2 (Equation 6)

[0085] Where d is the filter size and p is an adjustable parameter falling in (0, inf).

[0086] When p is small, segmentation module 204 produces a large but weak halo; when p is large, segmentation module 204 produces a strong expansion at the center. Figure 11 Example 1100 of a segmented mask with a specific p value is shown. When p = 0.5, the halo has weak expansion; when p = 2.0, the halo has strong expansion and the lines of the segmented mask are not as distinct as those of the segmented mask when p = 0.5.

[0087] In some embodiments, because blurring can blend within adjacent pixels and make power lines less noticeable, segmentation module 204 blurs around power lines to avoid situations where false alarms are more likely to occur, such as other cables or fences. The local blurring operation can be defined as:

[0088] f lb (I,M;g)=g(I)M+I(1-M) (Equation 7)

[0089] Where f lb is a function for local blurring, M is an extended electric field mask, and g is a blurring function (e.g., Gaussian). Figure 12 Example 1200 shows the original image and an image with local blurring and expanded segmented masking.

[0090] In some embodiments, the segmentation module 204 adds negative examples to the training. In some embodiments, the training data includes... 1 / 3 negative examples, and the classes include trees, fences, and buildings. Segmentation module 204 can increase the number and diversity of negative examples. Additional categories may include wire, steel rope, barbed wire, wire fence, picket fence, and fence post.

[0091] To avoid the segmentation module 204 generating segmentation masks for utility poles on non-pole vertical structures, the segmentation module 204 applies a heuristic during inference time. For example, this could cause the segmentation module 204 to remove utility poles instead of the power lines attached to them. In some embodiments, the segmentation module 204 runs a breadth-first search (BFS) to obtain connected components. Then, for each utility pole component, the segmentation module 204 calculates the shortest distance to the power line. The segmentation module 204 removes the entire pole object when the power line is too far away (this could be an adjustable parameter). Assuming N pixels and K components, BFS is O(N). If the segmentation module 204 iterates over pixel pairs, the shortest distance becomes the bottleneck, and the complexity becomes O(N). 2The segmentation module 204 can simplify the object to a rotated box, which reduces the complexity to O(K). 2 ).

[0092] In some embodiments, the segmentation module 204, stored on media server 101, is trained using training data to generate a high-capacity model. The segmentation module 204 then generates a low-capacity model by running inference on a much larger dataset segmented from the high-capacity model; this model includes a more compact neural network model. In this way, the high-capacity machine learning model is refined into a smaller, less resource-intensive, trained segmented machine learning model. The trained segmented machine learning model is used by the segmentation module 204, stored on user device 115. Because local operations such as blur and contrast are computationally expensive, in some embodiments, the segmentation module 204 pre-computes local operations and adds them to the dataset. Global operations (brightness) are computationally inexpensive, so the segmentation module 204 performs this analysis dynamically during training and adds randomness to achieve diversity.

[0093] A trained machine learning model can include one or more model forms or structures. For example, a model form or structure can include any type of neural network, such as linear networks, deep learning neural networks that implement multiple layers (e.g., "hidden layers" between the input and output layers, each of which is a linear network), convolutional neural networks (e.g., networks that split or divide input data into multiple parts or tiles, process each tile individually using one or more neural network layers, and aggregate the processing results for each tile), sequence-to-sequence neural networks (e.g., networks that receive sequential data as input—such as words in a sentence, frames in a video, etc.—and produce a sequence of results as output), and so on.

[0094] The model form or structure can specify the connectivity between various nodes and the organization of nodes within layers. For example, nodes in the first layer (e.g., the input layer) can receive data as input data or application data. Such data may include, for example, one or more pixels for each node, for example, when the trained model is used for, for example, the analysis of media items. Subsequent intermediate layers can receive the outputs of nodes in the previous layer as input, according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer may output segments between the foreground and background. The last layer (e.g., the output layer) produces the output of the machine learning model. For example, the output layer may receive segments of media items to the foreground and background and output whether a pixel is part of a segment mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0095] In various embodiments, the trained model may include one or more models. One or more models may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, a node may be a computation node without memory, for example, configured to process a unit of input to produce a unit of output. Computations performed by a node may include, for example, multiplying each of the multiple node inputs by a weight to obtain a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce a node output. In some embodiments, computations performed by a node may include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, computations are performed in parallel by multiple nodes, for example, using multiple processor cores of a multi-core processor, a single processing unit using a graphics processing unit (GPU), or a dedicated neural circuit. In some embodiments, a node may include memory, for example, to store and use one or more earlier inputs while processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. LSTM nodes can use memory to maintain a "state," which allows the node to operate like a finite state machine (FSM).

[0096] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may be started as multiple nodes organized into layers specified by a model form or structure. During initialization, appropriate weights may be applied to the connections between each pair of nodes connected according to the model form, such as nodes in consecutive layers of a neural network. For example, the appropriate weights may be randomly assigned or initialized to default values. The model can then be trained, for example, using training data to produce results.

[0097] Training can include the application of supervised learning techniques. In supervised learning, training data can include multiple inputs (e.g., manually annotated clips and synthesized media items) and a corresponding ground truth output for each input (e.g., a ground truth segment mask that correctly identifies one or more distracting objects). Based on a comparison between the model output and the ground truth output, weights are automatically adjusted, for example, to increase the probability that the model will generate ground truth outputs for the media items.

[0098] In some embodiments, during training, the segmentation module 204 outputs segmented masks and segmentation scores, which indicate the quality of the segmented masks used to identify power lines, poles, and / or towers in media items. The segmentation score may reflect the intersection-over-union (IoU) ratio between the segmented mask output by the segmented machine learning model and the ground-based segmented mask. In some embodiments, the segmentation score is a number in the total, such as 40 / 100. Other representations of the segmentation score are possible.

[0099] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights, for example, downloaded from a server providing the weights. In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmentation module 204 may include a previously trained model, for example, by the developer of the segmentation module 204, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights, for example, downloaded from a server providing the weights.

[0100] In some embodiments, segmentation module 204 receives media items. Segmentation module 204 provides media items as input to a trained machine learning model. In some embodiments, the trained machine learning model outputs segmented masks of one or more perturbation objects.

[0101] In some embodiments, the machine learning model outputs a confidence value for each segmented mask output by the trained machine learning model. Confidence can be expressed as a percentage, a number from 0 to 1, etc. For example, the machine learning model outputs an 85% confidence value for the segmented mask correctly covering one or more interfering objects.

[0102] In some embodiments, media items and segment masks are provided as input to the repair module 206.

[0103] Repair module 206 generates output media items in which one or more interfering objects are absent (erased from the source media item). In some embodiments, repair module 206 includes a set of instructions executable by processor 235 to generate the output media items. In some embodiments, repair module 206 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0104] In some embodiments, the repair module 206 receives a segmented mask from the segmentation module 204. The repair module 206 repairs a portion of a media item that matches the segmented mask. For example, the repair module 206 replaces all pixels within the segmented mask with pixels that match the background in the media item. In some embodiments, the pixels matching the background may be based on another media item at the same location. In some embodiments, the repair module 206 replaces pixels identified by the segmented mask with pixels that match adjacent regions within the media item.

[0105] In some embodiments, the repair module 206 trains a repair machine learning model to receive media items and segmentation masks from the segmentation module 204 as input, and outputs an output media item having one or more interfering objects that are not present in the output media item.

[0106] In some embodiments, the repair module 206 generates a repair confidence metric used to determine the quality of the output media item. In some embodiments, an evaluator (e.g., evaluating a machine learning model) is used to determine the quality of the output media item. Quality may be based on whether interfering objects were correctly removed (e.g., all interfering objects and related features, such as shadows, were removed) and whether the repair is indistinguishable (e.g., the repaired area of ​​the output media item blends in with the rest of the media item, such that removing interfering objects does not lead to perceptible inaccuracies).

[0107] User interface module 208 generates a user interface. In some embodiments, user interface module 208 includes a set of instructions executable by processor 235 to generate the user interface. In some embodiments, user interface module 208 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0108] User interface module 208 generates the user interface, which is executed by... Figure 2 Any steps and references performed by the modules in the document Figure 14 Before describing the steps, request permission from the user to access their media items.

[0109] User interface module 208 generates a user interface that includes media items and provides suggestions for removing one or more interfering objects. For example, the user interface may include an outline around each interfering object. In some embodiments, if an interfering object is detected to have a confidence threshold level, and if the quality of the output media item meets a quality threshold, the user interface module 208 generates a suggestion to automatically remove the interfering object. This ensures a satisfactory user experience for automatically removing interfering objects.

[0110] User interface module 208 generates a user interface that includes the repaired output media item. In some embodiments, the output media item may be annotated (visually) or marked (in code, e.g., implicitly) to indicate that the output media item is edited to erase one or more objects. In some embodiments, the user interface includes options for editing the output media item, sharing the output media item, adding the output media item to an album, etc. Options for editing the output media item may include the ability to undo object erasure.

[0111] In some embodiments, the user interface module 208 receives feedback from a user on the user device 115. The feedback may take the form of the user posting output media items, deleting output media items, sharing output media items, etc.

[0112] Example Method

[0113] Figure 13 A flowchart of an example method 1300 for training a segmented machine learning model is shown. Figure 13 Method 1300 can begin from block 1302. Method 1300 shown in the flowchart can be... Figure 2 The method is executed by the computing device 200. In some embodiments, the method 1300 is executed by the user device 115, the media server 101, or partly on the user device 115 and partly on the media server 101.

[0114] At box 1302, training data is generated, which includes a first group of media items and a second group of media items, wherein the first group of media items corresponds to the second group of media items and includes manually segmented noise objects. Box 1302 may be followed by box 1304.

[0115] At box 1304, a segmented machine learning model is trained based on training data to receive media items with one or more interfering objects and output segmented masks of one or more segmented objects corresponding to one or more interfering objects.

[0116] Figure 14 A flowchart of example method 1400 for removing interfering objects from a media item is shown. Figure 14 Method 1400 can begin from box 1402. Method 1400 shown in the flowchart can be... Figure 2 The method is executed by the computing device 200. In some embodiments, the method 1400 is executed by the user device 115, the media server 101, or partly on the user device 115 and partly on the media server 101.

[0117] At box 1402, receiving user permission can implement method 1400. For example, a user can load the application to view media items, but before displaying the media items, the user interface will request user permission to access the media items associated with the user. The user interface can also request permission to provide suggestions for media items, modify media items, enable the user to only allow access to specific media items, ensure that no media items are stored or transferred to the server without the user's permission, etc. Box 1402 can be followed by box 1404.

[0118] At box 1404, it is determined whether user permission has been received. If user permission has not been received, box 1406 follows box 1404, which stops method 1400. If user permission has been received, box 1408 follows box 1404.

[0119] At box 1408, media items are received from the user. For example, media items may include interfering objects such as power lines, utility poles, or towers. Box 1408 may be followed by box 1410.

[0120] At box 1410, one or more interfering objects are identified in the media item. Box 1410 may be followed by box 1412.

[0121] At box 1412, the media item is fed to the trained segmented machine learning model. Box 1412 can be followed by box 1414.

[0122] At box 1414, a segmented mask of one or more distracting objects in the media item is output using a trained machine learning model. Box 1414 can be followed by box 1416.

[0123] At box 1416, a portion of the media item that matches the segment mask is repaired to obtain the output media item, in which one or more interfering objects do not exist in the output media item.

[0124] Following the foregoing description, users may be given control over whether and when the systems, programs, or functions described herein can collect user information (e.g., information about the user's media items, including images and / or videos, social networks, social actions or activities, occupation, user preferences (e.g., relative to objects in an image) or the user's current location), and whether the user sends content or communications from the server. Furthermore, some data may be processed in one or more ways before storage or use to remove personally identifiable information. For example, a user's identity may be processed to the point that it is impossible to determine personally identifiable information for the user, or, where location information (such as city, zip code, or state level) is available, the user's geographic location may be generalized to the point that it is impossible to determine the user's specific location. Therefore, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0125] In the foregoing description, numerous specific details have been set forth for illustrative purposes in order to provide a thorough understanding of the specification. However, it will be apparent to those skilled in the art that the disclosure may be practiced without these specific details. In some cases, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the above embodiments may be described primarily with reference to a user interface and specific hardware. However, this embodiment can be applied to any type of computing device capable of receiving data and commands, as well as any peripheral device providing services.

[0126] The phrase "some embodiments" or "some examples" mentioned in the specification refers to specific features, structures, or characteristics described in relation to said embodiments or examples that may be included in at least one implementation described. The phrase "in some embodiments" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0127] A portion of the detailed description above is presented in the form of algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to those skilled in the art. Here, an algorithm is generally considered to be a self-consistent sequence of steps leading to a desired result. Steps are those steps that require physical operations on physical quantities. Typically, though not always, these quantities take the form of electrical or magnetic data capable of being stored, transmitted, combined, compared, and otherwise manipulated. It has proven convenient, sometimes, primarily for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0128] However, it should be remembered that all these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities. Unless it becomes apparent from the discussion below that a specific statement is made otherwise, it will be understood that throughout the description, the use of terms such as “processing” or “computing” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the computer system registers and memory and convert that data into other data similarly represented as physical quantities within the computer system memory or registers or other such information storage, transmission, or display devices.

[0129] The embodiments described in the specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor, selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including optical discs, ROMs, CD-ROMs, magnetic disks, RAM, EPROMs, EEPROMs, magnetic cards or optical cards, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0130] This specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments that include both hardware and software elements. In some embodiments, the specification is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0131] Furthermore, the description may take the form of a computer program product, accessible from a computer-usable or computer-readable medium, providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any means that may include, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0132] A data processing system suitable for storing or executing program code will include at least one processor that is directly or indirectly coupled to memory elements via a system bus. Memory elements may include local memory, mass storage, and cache memory used during the actual execution of the program code, which provide temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution.

Claims

1. A computer-implemented method for removing interfering objects from a media item, comprising: Generate training data including a first set of media items and a second set of media items, wherein the first set of media items includes interfering objects and the second set of media items includes manually segmented interfering objects; Identify one or more original media items in the first group of media items, which includes one or more disconnected power lines; Generate one or more corrected media items to correct the one or more disconnected power lines; One or more enhanced media items are generated by mixing portions of the one or more corrected media items with portions of the corresponding one or more original media items to increase the randomness of the enhancement; and The segmented machine learning model is trained based on the training data to receive media items with one or more interfering objects and output segmented masks of one or more segmented objects corresponding to the one or more interfering objects.

2. The method according to claim 1, wherein, Generating the one or more enhanced media items includes mixing the one or more corrected media items with corresponding one or more original media items with a checkerboard mask.

3. The method according to claim 1, wherein, Generating the one or more corrected media items to correct the one or more disconnected power lines includes: Modify the local contrast in one or more original media items to generate one or more corresponding enhanced media items.

4. The method according to claim 3, wherein, The local contrast is modified using a gain curve that combines the two bias curves.

5. The method according to claim 1, wherein, Generating the training data includes enhancing one or more media items in the first group of media items by applying an expansion to the segmented mask of the one or more disturbance objects.

6. The method according to claim 1, wherein, The one or more interfering objects are organized into categories, which include at least one group selected from power lines, utility poles, towers, and combinations thereof.

7. The method according to claim 1, wherein, Training the segmented machine learning model includes: Generate a high-capacity machine learning model based on the training data; and By running inference on the training data segmented by the high-capacity machine learning model, the high-capacity machine learning model is refined into a trained segmented machine learning model.

8. The method according to claim 1, wherein, The training data also includes a synthetic image in which the disturbance object is added in front of the outdoor environment object.

9. A computer-implemented method for removing interfering objects from a media item, the method comprising: Receive media items from the user; Identify one or more interfering objects in the media item; The media items are fed into a trained segmented machine learning model; Using the trained segmented machine learning model, output segmented masks of the one or more interfering objects in the media item; as well as Repair a portion of the media item that matches the segmented mask to obtain an output media item, wherein the one or more interfering objects do not exist in the output media item; The trained segmented machine learning model is trained by generating training data in the following way: Identify one or more original media items in a first group of media items that includes one or more disconnected power lines; Generate one or more corrected media items to correct the one or more disconnected power lines; and One or more augmented media items for the training data are generated by mixing portions of the one or more corrected media items with portions of the corresponding one or more original media items to increase the randomness of the augmentation.

10. The method according to claim 9, wherein, The one or more interfering objects are organized into categories, which include at least one group selected from power lines, utility poles, towers, and combinations thereof.

11. The method of claim 9, further comprising providing the user with a suggestion to remove the one or more interfering objects from the media item.

12. The method according to claim 9, wherein, The trained segmented machine learning model is trained using training data including the first set of media items and the second set of media items, wherein the first set of media items includes interfering objects and the second set of media items includes manual segmentation of the interfering objects.

13. A non-transitory computer-readable medium storing instructions thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform operations, the operations including: Generate training data including a first set of media items and a second set of media items, wherein the first set of media items includes interfering objects and the second set of media items includes manually segmented interfering objects; Identify one or more original media items in the first group of media items, which includes one or more disconnected power lines; Generate one or more corrected media items to correct the one or more disconnected power lines; One or more enhanced media items are generated by mixing portions of the one or more corrected media items with portions of the corresponding one or more original media items to increase the randomness of the enhancement; and The segmented machine learning model is trained based on the training data to receive media items with one or more interfering objects and output segmented masks of one or more segmented objects corresponding to the one or more interfering objects.

14. The computer-readable medium according to claim 13, wherein, Generating the one or more enhanced media items includes mixing the one or more corrected media items with corresponding one or more original media items with a checkerboard mask.

15. The computer-readable medium according to claim 13, wherein, Generating the one or more corrected media items to correct the one or more disconnected power lines includes: Modify the local contrast in one or more original media items to generate one or more corresponding enhanced media items.

16. The computer-readable medium of claim 15, wherein, The local contrast is modified using a gain curve that combines the two bias curves.

17. The computer-readable medium of claim 13, wherein, Generating the training data includes enhancing one or more media items in the first group of media items by applying an expansion to the segmented mask of the one or more disturbance objects.

18. The computer-readable medium according to claim 13, wherein, The one or more interfering objects are organized into categories, which include at least one group selected from power lines, utility poles, towers, and combinations thereof.

19. The computer-readable medium according to claim 13, wherein, Training the segmented machine learning model includes: Generate a high-capacity machine learning model based on the training data; and By running inference on the training data segmented by the high-capacity machine learning model, the high-capacity machine learning model is refined into a trained segmented machine learning model.

20. The computer-readable medium of claim 13, wherein, The training data also includes a synthetic image in which the disturbance object is added in front of the outdoor environment object.

Images