Detecting digital objects on a device and generating object masks

Abstract

A digital object is detected on a device and an object mask is generated. The present disclosure relates to systems, methods, and non-transitory computer-readable media that utilize a detection-masking neural network pipeline to generate object masks for digital objects depicted in digital images. In particular, in one or more embodiments, the disclosed systems utilize detection heads of a neural network to detect digital objects depicted in digital images. In some cases, each detection head is associated with one or more digital object classes that are unrelated to other detection heads. Furthermore, in some cases, the detection heads implement multi-scale synchronized batch normalization to normalize feature maps across various feature scales. The disclosed systems also utilize masking heads of the neural network to generate one or more object masks for the detected digital objects. In some cases, the disclosed systems utilize post-processing techniques to filter out low-quality masks.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 271,150, filed October 23, 2021, the entire contents of which are incorporated herein by reference. Background Technology

[0003] In recent years, significant progress has been made in hardware and software platforms for image editing. For example, many systems exist that can detect digital objects depicted in digital images. Furthermore, some conventional systems generate object masks for the digital objects detected in digital images. Using the generated object masks, conventional systems are able to perform various modifications to digital images (e.g., by removing or editing digital objects or applying filters to the background of the digital image).

[0004] However, despite these advancements, conventional image editing systems still suffer from several technical shortcomings that lead to inefficiencies, inflexibility, and inaccuracies. For example, many conventional systems utilize model components that require excessive computational resources, such as memory and processing power. Some conventional systems, for instance, employ group-normalized neural network layers or large backbone networks for object detection, which are computationally expensive during inference.

[0005] Beyond the efficiency issues mentioned above, conventional image editing systems often lack operational flexibility. For example, due to the significant consumption of computational resources, many conventional systems are strictly limited to relying on resource-rich devices to perform object detection and / or segmentation. To illustrate, many conventional systems require client devices displaying digital images to offload object detection and segmentation operations to another device (e.g., a server), as client devices lack the resources necessary to implement computationally expensive models for such operations. Furthermore, some conventional systems are strictly limited to training examples with a closed set of predefined category labels when learning to identify and segment objects. As a result, many conventional systems are prevented from detecting / classifying and then segmenting a wide range of object instances in digital images not included in the training examples.

[0006] Other conventional image editing systems often operate inaccurately. For example, by limiting training to a closed set of examples, conventional systems typically learn only category-specific features associated with that closed set and fail to generalize to objects of unseen categories. Therefore, conventional systems cannot accurately detect digital objects they have not yet seen. Furthermore, conventional systems may use tight bounding boxes for detected objects; however, such bounding boxes can be detrimental to segmentation, often resulting in artifacts where object pixels appear unnaturally glued to the box boundaries.

[0007] Conventional image editing systems have these and additional problems and challenges. Summary of the Invention

[0008] Systems, methods, and non-transitory computer-readable media that utilize efficient neural network models to implement flexible on-device pipelines for object detection and segmentation, one or more embodiments described herein, provide benefits and / or solve one or more of the aforementioned problems in the art. Specifically, in one or more embodiments, the system employs a neural network model having a detection head for identifying objects within a digital image and a masking head for generating object masks for the identified objects. In some cases, the neural network model includes multiple detection heads that collectively perform category-agnostic object detection. In other cases, the detection heads utilize multi-scale synchronous batch normalization to perform object detection. In some implementations, the system further utilizes post-processing techniques to filter out low-quality object masks. In this way, the system achieves an efficient and lightweight model capable of performing object detection and segmentation on-device while ensuring mask quality.

[0009] Additional features and advantages of one or more embodiments of this disclosure are outlined in the following description and will be apparent in part from the description, or may be learned by practice of such exemplary embodiments. Attached Figure Description

[0010] This disclosure will describe one or more embodiments of the invention with additional specificity and detail by referring to the accompanying drawings. The following paragraphs briefly describe these drawings, in which:

[0011] Figure 1 An example environment in which a masking system operates on a device according to one or more embodiments is shown;

[0012] Figures 2A to 2E An overview diagram of an on-device masking system, according to one or more embodiments, is shown;

[0013] Figures 3A to 3B A visual comparison of detection results obtained through category-specific object detection and category-agnostic object detection, according to one or more embodiments, is shown;

[0014] Figures 4A to 4D A neural network architecture for a detection masking neural network, according to one or more embodiments, is illustrated, which is utilized by a masking system on an on-device to perform object detection and mask generation.

[0015] Figures 5A to 5BA visual comparison of approximate boundaries and extended approximate boundaries generated for digital objects according to one or more embodiments is shown;

[0016] Figure 6 A block diagram for training a detection masking neural network according to one or more embodiments is shown;

[0017] Figures 7A to 7C A visual comparison of object details captured by an object mask and an upsampled object mask according to one or more embodiments is shown;

[0018] Figure 8 A block diagram is shown for determining a set of object masks for rendering using post-processing techniques, according to one or more embodiments;

[0019] Figure 9 An example schematic diagram of a masking system on a device according to one or more embodiments is shown;

[0020] Figure 10 A flowchart illustrating a series of actions for generating an object mask for a digital object in a digital image using a detection masking neural network, according to one or more embodiments, is shown; and

[0021] Figure 11 A block diagram of an exemplary computing device according to one or more embodiments is shown. Detailed Implementation

[0022] One or more embodiments described herein include an on-device masking system that utilizes an efficient and flexible neural network model for on-device object detection and instance segmentation. For illustration, in one or more embodiments, the on-device object detection system implements a neural network having one or more detection heads for object detection. In some cases, each detection head learns to detect objects associated with one or more digital object categories that are different from digital object categories learned by other detection heads. Furthermore, the neural network utilizes the detection heads(s) to perform category-agnostic object detection. In some implementations, the neural network also includes a masking head that generates an object mask for the detected digital objects. Thus, in some embodiments, the on-device masking system utilizes a neural network as a pipeline to receive digital images, detect objects depicted by the digital images, and generate an object mask for each detected object on a host computing device (e.g., without relying on additional devices such as servers).

[0023] For illustration, in one or more embodiments, the on-device masking system receives a digital image at a computing device (e.g., a computing device on which the on-device masking system operates). The on-device masking system at the computing device utilizes a detection head of a detection masking neural network to detect digital objects depicted in the digital image. Furthermore, the on-device masking system at the computing device utilizes a masking head of the detection masking neural network to generate an object mask for the digital objects.

[0024] As just mentioned, in one or more embodiments, the on-device masking system utilizes a detection masking neural network to provide a pipeline for detecting digital objects within a digital image and generating an object mask for those digital objects. For example, in some cases, the detection masking neural network includes a backbone network (e.g., a neural network encoder) for extracting features from the digital image and a series of function-based heads (e.g., a neural network decoder head) for detecting digital objects and generating the object mask. In practice, in one or more embodiments, the detection masking neural network includes a detection head that detects digital objects within the digital image. In some cases, the detection masking neural network includes more than one detection head.

[0025] For illustration, in some implementations, the detection masking neural network includes a multi-branch design with multiple detection heads that learn to detect digital objects from multiple digital image datasets. Specifically, in some embodiments, each detection head learns from a corresponding digital image dataset, which is different from the digital image datasets used for other detection heads. Furthermore, in some cases, each digital image dataset is associated with one or more digital object categories, which are different from the digital object categories associated with other digital image datasets. Therefore, the detection masking neural network learns to detect digital objects from a broad range of object categories.

[0026] Furthermore, in one or more embodiments, the detection masking neural network utilizes (multiple) detection heads to perform category-agnostic object detection. For example, in some cases, the detection masking neural network utilizes (multiple) detection heads to generate an objectivity score that indicates whether a portion of a digit image corresponds to a general object or to the background of the digit image.

[0027] In one or more embodiments, each detection head of the detection masking neural network uses at least one multi-scale batch normalized neural network layer to detect digital objects within a digital image. On-device masking systems utilize multiple multi-scale batch normalized neural network layers to facilitate on-device implementation of detection masking neural networks by reducing computational requirements.

[0028] Furthermore, in some embodiments, the detection masking neural network includes a masking head that generates an object mask for digital objects detected from a digital image. For example, in some implementations, the detection head identifies a plurality of digital objects depicted in the digital image, and the masking head generates at least one object mask for each of the identified digital objects. In some embodiments, the detection masking neural network generates extended approximate boundaries (e.g., bounding boxes or coarse boundary depictions) corresponding to the identified digital objects, and the masking head uses the extended approximate boundaries to generate the object mask.

[0029] In one or more embodiments, the on-device masking system implements one or more post-processing techniques to determine which object masks from the object masks generated by the detection masking neural network will be rendered for display on a computing device (e.g., a host computing device). For example, in some cases, the on-device masking system determines a mask quality score for the object masks generated by the detection masking neural network. Furthermore, in some cases, the on-device masking system utilizes the mask quality score to determine a set of object masks to be used for rendering via the computing device. Specifically, in some cases, the on-device masking system determines to exclude one or more object masks from rendering based on the corresponding mask quality score of the object mask and / or other associated attributes.

[0030] On-device masking systems offer several advantages over conventional systems. For example, they operate with greater efficiency compared to conventional systems. In particular, by implementing multi-scale synchronous batch normalization in the detection head of the detection masking neural network, on-device masking systems reduce the number of computations required during inference compared to many conventional systems that include group normalization. Therefore, on-device masking systems are configured for on-device inference and have a smaller computational footprint compared to conventional object detection and masking systems.

[0031] Furthermore, on-device masking systems offer greater operational flexibility compared to conventional systems. For example, by reducing computational resource consumption, on-device masking systems can flexibly perform object detection and instance segmentation on the device itself. In other words, on-device masking systems enable host computing devices to avoid reliance on additional computing devices (e.g., servers) for these operations. Moreover, as the detection heads of the detection masking neural network learn to detect objects from multiple digital image datasets associated with multiple object categories, on-device masking systems flexibly detect digital objects from a wide range of object instances.

[0032] Furthermore, on-device masking systems operate more accurately compared to conventional systems. To illustrate, by utilizing a wider variety of digital images during learning to detect digital objects, on-device masking systems more accurately detect digital objects that have not been explicitly learned. Moreover, by generating extended approximate boundaries corresponding to the identified digital objects, on-device masking systems produce more accurate object masks without the boundary artifacts seen in many conventional systems.

[0033] Furthermore, the on-device masking system exhibits improved generalization and accuracy strength. Specifically, the on-device masking system utilizes a training strategy that adaptively selects positive and negative samples based on their statistical properties. Moreover, to further enhance the generalization strength of the on-device masking system, a multi-dataset training scheme (see description below) forces the on-device masking system to perform well on multiple object detection and instance segmentation datasets simultaneously. To improve mask quality, in one or more implementations, the on-device masking system performs one or more of the following: upsampling mask predictions during training to benefit from higher resolution and higher quality ground truth masks; artificially expanding the detection boxes used by the masking head to avoid boundary artifacts; utilizing parallel dilated convolutional layers with different dilation rates for high-resolution mask predictions; and filtering out less accurate mask predictions during post-processing using mask quality predictions.

[0034] In addition to the foregoing, conventional object detection and instance segmentation models are expected to predict labels based on a predefined set of category labels. By training on data with specific category labels, the model can only learn category-specific features, which are useful for detecting objects belonging to such category labels but do not generalize well to objects in unseen categories. To address this issue, on-device masking systems transform the conventional category-specific object detection task into a category-agnostic task. Instead of having the model learn to predict specific category labels, on-device masking systems treat all detected ground truth samples as a single object category and distinguish the single object (positive) category from the background (negative) category. This allows on-device masking systems to learn general objectivity (common properties shared by various objects) rather than focusing on fine-grained visual details used to distinguish objects from different category labels. Therefore, in one or more implementations, on-device masking systems offer increased flexibility compared to conventional systems. For example, on-device masking systems are not limited to the types of objects seen during training. Instead, on-device masking systems expose the segmentation of all objects (and optional object parts), regardless of the classification type or whether the object (or object part) was seen during training.

[0035] As discussed above, this disclosure utilizes various terms to describe the features and benefits of the masking system on the device. Further details regarding the meaning of these terms are now provided. For example, as used herein, the term "digital object" refers to a distinguishable element depicted in a digital image. For illustration, in some embodiments, a digital object includes people, items, natural objects (e.g., trees or rock formations), or structures depicted in a digital image. In some cases, a digital object refers to multiple elements that can be distinguished from other elements depicted in a digital image. For example, in some cases, a digital object includes a collection of buildings that constitute a skyline. In some cases, a digital object more broadly includes (is part of) the foreground or other elements distinct from the background depicted in a digital visualization media item.

[0036] In one or more embodiments, a digital object is associated with at least one digital object category. As used herein, the term "digital object category" refers to a classification or type of digital object. Specifically, in some embodiments, a digital object category refers to a grouping of digital objects based on one or more attributes common to the included digital objects. For illustration, in some cases, a digital object category includes, but is not limited to, categories corresponding to dogs, cats, people, cars, boats, birds, buildings, fruits, telephones, or computer equipment. The generalization of a digital object category regarding the digital objects it includes varies in different embodiments. Relatedly, as used herein, the term "category label" refers to a mark or tag associated with a digital object based on a corresponding digital object category.

[0037] As used herein, the term "digital image dataset" includes a collection of digital images. Specifically, in some embodiments, a digital image dataset includes a set of digital images depicting at least one digital object. In some implementations, a digital image dataset corresponds to one or more digital object categories. For example, in some cases, the digital images in a digital image dataset depict at least one digital object corresponding to a common digital object category (e.g., associated with a common category label).

[0038] Furthermore, as used herein, the term "object mask" refers to a boundary line used to divide a digital image into separate portions. Specifically, in some embodiments, an object mask refers to the identification of portions of a digital image belonging to one or more digital objects (i.e., pixels of the digital image) and portions of a digital image belonging to the background and / or other objects. For example, in some embodiments, an object mask includes a mapping of the digital image having an indication for each pixel regarding whether that pixel corresponds to a portion of a digital object. In some implementations, this indication includes a binary indication (e.g., "1" indicates a pixel belonging to a digital object, and "0" indicates a pixel not belonging to a digital object). In alternative implementations, the indication includes a probability indicating the likelihood that a pixel belongs to a digital object (e.g., a number between 1 and 0). In such implementations, the closer the value is to 1, the more likely the pixel belongs to a digital object, and vice versa.

[0039] As used herein, the term "neural network" refers to a machine learning model that can be tuned (e.g., trained) based on inputs to approximate an unknown function used to generate a corresponding output. Specifically, in some embodiments, a neural network refers to a model of interconnected artificial neurons (e.g., hierarchically organized) that communicate and learn to approximate complex functions and generate outputs based on multiple inputs provided to the model. In some cases, a neural network includes one or more machine learning algorithms. Furthermore, in some cases, a neural network includes algorithms (or sets of algorithms) that implement deep learning techniques that utilize the set of algorithms to model high-level abstractions in data. For illustration, in some embodiments, a neural network includes convolutional neural networks, recurrent neural networks (e.g., long short-term memory neural networks), generative adversarial neural networks, graph neural networks, or multilayer perceptrons. In some embodiments, a neural network includes a neural network or a combination of neural network components.

[0040] Furthermore, as used herein, the term "detection masking neural network" refers to a computer-implemented neural network that generates object masks for digital objects depicted in the digital image. Specifically, in some embodiments, a detection masking neural network refers to a computer-implemented neural network that detects digital objects within a digital image and generates object masks for those digital objects. In practice, in some implementations, a detection masking neural network includes a neural network pipeline that analyzes the digital image, identifies one or more digital objects depicted in the digital image, and generates object masks for the one or more digital objects.

[0041] Furthermore, as used herein, the term "detection head" refers to a neural network component that detects digital objects depicted in a digital image. Specifically, in some embodiments, a detection head refers to one or more sets of neural network layers that detect digital objects depicted in a digital image. For example, in some cases, a detection head refers to a neural network decoder that detects digital objects from a digital image based on the encoding of the digital image. In some implementations, the detection head detects digital objects corresponding to one or more digital object categories.

[0042] As used herein, the term "mask head" refers to a neural network component that generates an object mask for a digital object. For example, in some cases, a mask head refers to one or more sets of neural network layers that generate an object mask for a digital object. For example, in some cases, a mask head refers to a neural network decoder that generates an object mask for a digital object identified from a digital image.

[0043] Furthermore, as used herein, the term "parameter" refers to a variable within a computer-implemented model (such as a detection masking neural network). Specifically, in some embodiments, a parameter refers to a variable that affects the operation of the corresponding computer-implemented model. For example, in some cases, parameters include weights of a function of the computer-implemented model that affects the results generated by the model. In some cases, parameters are associated with a specific component of the computer-implemented model. For example, in some implementations, parameters are associated with the detection head or masking head of a detection masking neural network. Relatedly, as used herein, the term "convolution parameter" refers to a parameter corresponding to a convolutional layer of a neural network. Similarly, as used herein, the term "output parameter" refers to a parameter corresponding to the output layer or a component of a neural network. For example, in some cases, output parameters correspond to the output layer of the detection head or masking head of a detection masking neural network.

[0044] Furthermore, as used herein, the term "multi-scale synchronous batch normalization neural network layer" refers to a neural network layer that performs batch normalization across various feature maps. Specifically, in some embodiments, a multi-scale synchronous batch normalization neural network layer refers to a neural network layer that performs synchronous batch normalization on feature maps corresponding to image features at various levels. For example, in some cases, a multi-scale synchronous batch normalization neural network layer includes a neural network layer that simultaneously normalizes features across two dimensions: multiple pyramid levels of the feature maps and multiple graphics processing units (GPUs).

[0045] As used herein, the term "objectivity score" refers to a value indicating whether a digital image depicts a digital object. Specifically, in some embodiments, the objectivity score refers to a score or other quantifiable value that corresponds to a portion of a digital image and indicates whether that portion of the digital image includes a digital object. In one or more embodiments, the objectivity score is category-agnostic (e.g., a "category-agnostic objectivity score"). In other words, the objectivity score indicates whether a portion of a digital image corresponds to a general digital object (e.g., not a digital object of a specific object category) or to the background.

[0046] Furthermore, as used herein, the term "approximate boundary" refers to the boundary of a digital object within an approximate digital image. Specifically, in some embodiments, an approximate boundary refers to the boundary (e.g., boundary delineation) of the outline or outer edge of a digital object depicted within a digital image. Alternatively, in some cases, an approximate boundary refers to a bounding box or some other shape surrounding the digital object. In some implementations, the approximate boundary closely follows the shape of the digital object's outline. Relatedly, as used herein, the term "extended approximate boundary" refers to an approximate boundary that has been generated from another approximate boundary and is larger than the other approximate boundary in at least one dimension.

[0047] Additional details regarding the shielding system on the equipment will now be provided with reference to the accompanying drawings. For example, Figure 1 A schematic diagram of an exemplary system 100 is shown, in which a masking system 106 (also optionally referred to herein as a DOMO or detection object mask object) is on-device. Figure 1 As shown, system 100 includes (multiple) servers 102, network 108, and client devices 110a to 110n.

[0048] Although Figure 1 System 100 is described as having a specific number of components, but system 100 can have any number of additional or alternative components (e.g., any number of servers, client devices, or other components communicating with the masking system 106 on the device via network 108). Similarly, although Figure 1 A specific arrangement of (multiple) servers 102, network 108, and client devices 110a to 110n is shown, but various additional arrangements are possible.

[0049] Multiple servers 102, network 108, and client devices 110a to 110n communicate and are coupled to each other directly or indirectly (e.g., through the following...). Figure 11 (Network 108 is discussed in more detail). Furthermore, the (multiple) servers 102 and client devices 110a to 110n include computing devices among various computing devices (including, as per...). Figure 11 (One or more computing devices discussed in more detail).

[0050] As described above, system 100 includes servers 102. In one or more embodiments, servers 102 generate, store, receive, and / or transmit data, including neural networks, digital images, object masks, and modified digital images. In one or more embodiments, servers 102 include data servers. In some implementations, servers 102 include communication servers or web hosting servers.

[0051] In one or more embodiments, image editing system 104 provides users (e.g., users of one of client devices 110a to 110n) with the ability to generate, edit, manage, and / or store digital images. For example, in some cases, a user uses a client device to send a digital image to image editing system 104 hosted on server(s)102 via network 108. Image editing system 104 then provides a number of options that the user can use to edit the digital image, store the digital image, and subsequently search, access, and view the digital image.

[0052] In one or more embodiments, client devices 110a to 110n include computing devices capable of accessing, editing, segmenting, modifying, storing, and / or providing digital images for display. For example, client devices 110a to 110n include smartphones, tablets, desktop computers, laptops, head-mounted displays, or other electronic devices. Client devices 110a to 110n include one or more applications (e.g., image editing application 112) capable of accessing, editing, segmenting, modifying, storing, and / or providing digital images for display. For example, in one or more embodiments, image editing application 112 includes software applications installed on client devices 110a to 110n. Additionally or alternatively, image editing application 112 includes software applications hosted on server(s) 102 (and supported by image editing system 104), which client devices 110a to 110n can access via another application (such as a web browser).

[0053] To provide an illustrative implementation, in some embodiments, the on-device masking system 106 on server(s)102 supports the on-device masking system 106 on client device(s)110n. For example, in some cases, the on-device masking system 106 on server(s)102 learns parameters for a detection masking neural network. The on-device masking system 106 then provides the detection masking neural network to client device(s)110n via server(s)102. In other words, client device(s)110n obtains (e.g., downloads) the detection masking neural network with the learned parameters from server(s)102. Once downloaded, the on-device masking system 106 on client device(s)110n independently of server(s)102 uses the detection masking neural network to detect digital objects and generate corresponding object masks.

[0054] In an alternative implementation, the on-device masking system 106 includes a web-hosted application that allows a client device 110n to interact with content and services hosted on servers(s)102. For illustration, in one or more implementations, the client device 110n accesses web pages supported by servers(s)102. For example, the client device 110n provides a digital image to servers(s)102, and in response, the on-device masking system 106 on servers(s)102 generates one or more object masks. Server(s)102 then provides the object masks to the client device 110n for displaying or editing the digital image.

[0055] In fact, the masking system 106 on the device can be implemented wholly or partially by the individual elements of system 100. Although Figure 1 An on-device masking system 106 implemented with respect to (multiple) servers 102 is shown, but different components of the on-device masking system 106 can be implemented by various devices within system 100. For example, one or more (or all) components of the on-device masking system 106 may be implemented by different computing devices (e.g., one of client devices 110a to 110n) or by servers separate from (multiple) servers 102 hosting the image editing system 104. In fact, as Figure 1 As shown, client devices 110a to 110n include on-device masking systems 106. The following will discuss... Figure 9 Describe an example component of the masking system 106 on the device.

[0056] In one or more embodiments, the on-device masking system 106 operates to enable the selection of digital objects depicted within a digital image and the viewing of corresponding object masks. Specifically, as described above, in some cases, the on-device masking system 106 detects digital objects depicted in a digital image and generates corresponding object masks (e.g., on a host computing device such as a client device). Furthermore, in some embodiments, the on-device masking system 106 provides object masks for digital objects in response to selection of the digital objects via the computing device. Figures 2A to 2E An overview diagram of an on-device masking system 106, according to one or more embodiments, is shown. It facilitates the provision of an object mask corresponding to a selected digital object.

[0057] like Figure 2A As shown, the on-device masking system 106 provides a digital image 208 for display within a graphical user interface 204 of a computing device 206 (e.g., a client device). Furthermore, the on-device masking system 106 provides selectable options 202 for an automatic masking tool within the graphical user interface 204. In one or more embodiments, in response to receiving a user selection of the selectable option 202, the on-device masking system 106 detects digital objects depicted within the digital image 208 and generates a corresponding object mask for the detected digital objects, as will be explained in more detail below. In some embodiments, the on-device masking system 106 detects and generates object masks on the computing device 206 (e.g., without relying on the computing resources of a separate device such as a server to perform these operations).

[0058] like Figure 2B As shown, in response to the detection of a selection of a digital object 220 (e.g., a lemon) depicted within a digital image 208, the on-device masking system 106 provides an object mask 222 for display of the digital object 220. For example, in some cases, the on-device masking system 106 detects a selection of the digital object 220 by hovering a cursor 224 over the digital object 220, clicking the cursor 224 while positioned over the digital object 220, or a short / tap gesture of the digital object 220, and then provides the corresponding object mask 222.

[0059] Figure 2C As shown, in response to the detection of a selection of another digital object 240 depicted within the digital image 208 (e.g., movement of the cursor 224 towards the other digital object 240 or some other gesture indicating selection of the other digital object 240), the on-device masking system 106 provides a corresponding object mask 242 for display. Furthermore, the on-device masking system 106 removes the object mask 222 corresponding to the digital object 220. Similarly, as Figure 2DAs shown, in response to the selection of digital object 260 (e.g., movement toward digital object 260 via cursor 224), on-device masking system 106 provides object mask 262 corresponding to the third digital object 260 depicted in digital image 208.

[0060] Figure 2E It is also shown that the on-device masking system 106 provides an object mask 282 corresponding to the selected digital object 280 in response to the detection of the digital object 280. However, it should be noted that the digital object 280 (bowl) is different from the digital objects 220, 240, 260 (lemon) depicted in the digital image 208. In fact, as will be explained in more detail below, the on-device masking system 106 detects a wide range of digital objects depicted in the digital image and generates their corresponding object masks. Furthermore, in some cases, the on-device masking system 106 detects each digital object depicted in the digital image and generates a corresponding object mask (e.g., performing panorama segmentation).

[0061] As described above, the on-device masking system 106 generates an object mask for digital objects depicted in a digital image using very little (if any) user input. In fact, in some cases, the on-device masking system 106 performs object detection and mask generation on digital objects in the digital image before the user selects an object from the digital objects. Although the above discussion indicates that object detection and mask generation occur in response to the selection of an automatic masking tool, in one or more implementations, the on-device masking system 106 performs object detection and mask generation without such a selection (e.g., in response to receiving a digital image), which further reduces the required user input.

[0062] As previously described, in one or more embodiments, the on-device masking system 106 detects a wide range of digital objects within a digital image. Specifically, the on-device masking system 106 detects digital objects associated with a wide range of object categories. In some embodiments, the on-device masking system 106 implements category-agnostic object detection to detect digital objects within a digital image, regardless of their associated digital object category. Figures 3A to 3B A visual comparison is shown of the detection results obtained by category-specific object detection performed by multiple conventional systems and category-agnostic object detection performed by on-device masking system 106, according to one or more embodiments.

[0063] like Figure 3AAs shown, category-specific object detection results in the detection of digital object 304a in digital image 302, which corresponds to a specific digital object category (e.g., smartphone or handheld device category). In some cases, category-specific object detection results in the detection of multiple digital objects in a digital image, but these digital objects are all associated with a predefined set of digital object categories. In practice, many conventional systems learn a classifier that classifies regions of a digital image into one of N+1 digital object categories, where... This is the number of predefined category labels, and 1 corresponds to the background category. However, as... Figure 3A As shown, if a digital image depicts other digital objects that are not associated with the N+1 digital object categories (e.g., digital objects 304b, 304c, 304d, 304e, 304f, 304g), then category-specific object detection will typically fail to detect these objects.

[0064] By comparison, the on-device masking system 106 classifies regions of a digital image into one of two categories: an object category and a background category. To achieve this, the on-device masking system 106 treats all detected ground truth samples as a single object category and enables the detection masking neural network to learn to distinguish the object category from the background category using general object-specific features (e.g., common properties shared by all types of digital objects), as will be discussed in more detail below. Thus, the on-device masking system 106 transforms conventional category-specific object detection into a category-agnostic detection task. Figure 3B As shown in the detection results, by using a category-agnostic method for object detection, the masking system 106 on the device detects objects 304a to 304g within a digital image, regardless of their associated digital object category.

[0065] By implementing category-agnostic (object-aware) detection, the masking system 106 on the device operates more flexibly than conventional systems. In fact, as through... Figures 3A to 3B The comparison of the detection results shown demonstrates that, compared to many conventional systems, the on-device masking system 106 flexibly detects digital objects associated with a wider range of digital object categories. Therefore, in many cases, the on-device masking system 106 flexibly detects more digital objects within a digital image.

[0066] As described above, the masking system 106 on the device uses a detection masking neural network to detect digital objects within a digital image and generate an object mask for the detected digital objects. Figures 4A to 4D A detection masking neural network, according to one or more embodiments, is illustrated and utilized by a masking system 106 on an on-device to perform object detection and mask generation. Specifically, Figure 4A This provides a broad overview of the architecture of a detection masking neural network. Figures 4B to 4D Further details regarding the architecture and connectivity of various components of a detection masking neural network, according to one or more embodiments, are shown.

[0067] In fact, such as Figure 4A As shown, the detection masking neural network 400 includes a neural network encoder 402 with a backbone network, a detection head 404 (or a neural network decoder head), and a masking head 406 (or a neural network decoder head). Figure 4A As shown, the neural network encoder 402 encodes the digital image and provides the encoding to the detection head 404 and the masking head 406. The detection head 404 uses the encoding to detect one or more digital objects depicted in the digital image. The masking head 406 generates at least one object mask for each of the detected digital objects.

[0068] Figure 4B A neural network encoder 402, utilizing a detection masking neural network 400, is illustrated according to one or more embodiments to encode a digital image. For example... Figure 4B As shown, the neural network encoder 402 includes a backbone network that analyzes the digital image 410. Based on the analysis, the neural network encoder 402 generates a feature pyramid 412 (e.g., multiple feature maps at different resolutions, such that the feature maps capture image features from different levels of the digital image 410).

[0069] For example, as shown in the figure, the neural network encoder 402 generates feature maps 414a, 414b, and 414c from the digital image 410. The neural network encoder 402 uses feature maps 414a to 414c to generate feature maps 416a to 416e within the feature pyramid 412. For example, in some embodiments, the neural network encoder 402 uses convolutional layers with top-down connections to directly generate feature maps 416a, 416b, and 416c from feature maps 414a to 414c, respectively. In some cases, the neural network encoder 402 uses a convolutional layer with a stride of 2 to generate feature map 416d from feature map 416c. Similarly, in some cases, the neural network encoder 402 uses a convolutional layer with a stride of 2 to generate feature map 416e from feature map 416d.

[0070] like Figure 4B As shown, the neural network encoder 402 provides the feature maps 416a to 416e of the feature pyramid 412 to the detection head 404 for detecting digital objects within the digital image 410, and provides them to the masking head 406 for generating object masks for the detected digital objects.

[0071] Figure 4BTable 418 illustrates the architecture of a neural network encoder 402 according to one or more embodiments (e.g., the architecture for generating feature maps 414a to 414c). For example, in one or more embodiments, the on-device masking system 106 utilizes the EfficientNet-B0 backbone network as the neural network encoder 402, which is described by Mingxing Tan and Quoc Le in “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks” (International Conference on Machine Learning, pp. 6105–6114, PMLR, 2019).

[0072] By utilizing the neural network encoder 402 described above to generate feature maps from digital images, the on-device masking system 106 operates more efficiently than many conventional systems. In fact, the neural network encoder 402 comprises a smaller computational footprint compared to the backbone networks used in many conventional systems (e.g., ResNet-50 networks). Therefore, using the neural network encoder 402 reduces the computational resources required to generate feature maps from digital images. This further improves flexibility, as the reduced computation allows the on-device masking system 106 to be deployed on computing devices with limited resources (e.g., client devices).

[0073] Figure 4C The architecture of a detection head 404 of a detection masking neural network 400 according to one or more embodiments is shown. Figure 4C As shown, the detection head 404 includes multiple neural network layers. Further, the detection head 404 includes multiple output layers 426, with each detection head targeting one output layer. Similarly, the detection head 404 includes multiple multi-scale synchronous batch normalized neural network layers 428a to 428b, with each detection head targeting one or more multi-scale synchronous batch normalized neural network layers.

[0074] In fact, in one or more embodiments, the detection masking neural network 400 shares one or more neural network layers between the detection heads 404. Figure 4CA detection head sharing intermediate convolutional neural network layers and leakage correction linear unit (ReLU) neural network layers is shown. Sharing these layers among detection heads 404 allows the detection masking neural network 400 to remain lightweight, as these are computationally demanding layers that, in some cases, constitute most of the parameters for the detection head. In some implementations, sharing neural network layers among detection heads includes sharing the parameters of the neural network layers among the detection heads.

[0075] Additionally, in one or more embodiments, each detection head 404 includes a separate set of multi-scale synchronous batch normalized neural network layers 428a to 428b. In other words, the on-device masking system 106 learns individual parameters for different detection heads. In practice, given a number of channels... The batch normalization layer includes 2C learnable parameters, while a commonly used 3×3 convolutional neural network layer learns C×C×3×3 learnable parameters. Therefore, the multi-scale synchronous batch normalization neural network layer is relatively lightweight, and the on-device masking system 106 utilizes a separate multi-scale synchronous batch normalization neural network layer for each detection head without significantly increasing the computational footprint of the detection masking neural network 400.

[0076] In one or more embodiments, the detection head 404 utilizes multi-scale synchronous batch normalization neural network layers 428a to 428b to normalize feature maps across all levels of the feature pyramid generated by the neural network encoder 402 and across all GPUs. In practice, in some implementations, the detection head 404 utilizes multi-scale synchronous batch normalization neural network layers 428a to 428b to normalize features simultaneously across two dimensions.

[0077] As an example, in one or more embodiments, the detection masking neural network 400 utilizes multi-scale synchronous batch normalization neural network layers 428a to 428b across multiple GPUs to compute normalized statistics (mean and variance) from data samples. To illustrate, during parameter learning (e.g., during training), the detection head 404 utilizes the multi-scale synchronous batch normalization neural network layers 428a to 428b to compute the global mean (denoted as M) and the squared mean (denoted as P), as follows:

[0078]

[0079]

[0080] In Equations 1 and 2, |G| represents the number of GPUs, and |S| represents the number of layers in the feature pyramid generated by the neural network encoder 402. Additionally, This represents the scalar average of a single feature channel computed using the i-th GPU and the j-th pyramid level. Multi-scale synchronous batch normalization neural network layers 428a to 428b determine the global variance V = P - MM and normalize the feature mapping across the pyramid level using the global mean (M) and global variance (V). In one or more embodiments, the multi-scale synchronous batch normalization neural network layers 428a to 428b use a single set of running statistics for each feature channel.

[0081] Therefore, during inference, the detection masking neural network 400 maintains full parameter shareability across all pyramid levels of the detection head. Thus, while in some embodiments the detection masking neural network 400 learns a separate multi-scale synchronous batch normalization neural network layer for each detection head, it utilizes a multi-scale synchronous batch normalization neural network layer for multiple pyramid levels of the detection head.

[0082] By utilizing multi-scale synchronous batch normalization neural network layers, the on-device masking system 106 operates more efficiently than many conventional systems. Indeed, as previously mentioned, many conventional systems utilize group normalization for object detection, which introduces additional computational operations. Therefore, utilizing multi-scale synchronous batch normalization reduces the computational cost required for object detection while allowing for multi-scale operation. This increased efficiency facilitates the flexible deployment of the on-device masking system 106 on computing devices with relatively limited resources.

[0083] Furthermore, as described above, in one or more embodiments, each detection head 404 includes a separate output layer (e.g., the last layer of the detection head). In practice, as previously discussed and will be discussed in more detail below, in some implementations, the on-device masking system 106 utilizes separate digital image datasets to learn parameters for the detection heads 404. Therefore, by keeping the output layers 426 separate for each detection head, the detection masking neural network 400 maintains neural network layers tuned from a specific digital image dataset (the digital image dataset corresponding to the detection head of the output layer). Furthermore, the detection masking neural network 400 ensures that it learns distinct output spaces that exist independently of each other.

[0084] As described above, the detection masking neural network 400 utilizes a detection head 404 to detect digital objects 422a, 422b, and 422c within the digital image 410. Specifically, in one or more embodiments, the detection head 404 utilizes feature maps from a feature pyramid generated by the neural network encoder 402 to detect digital objects 422a to 422c. In some cases, utilizing a feature pyramid enables the detection head 404 and other detection heads to detect objects of different sizes depicted in the digital image. For example, in some cases, the detection head 404 utilizes the minimum feature map from the feature pyramid to detect relatively large objects and the maximum feature map (e.g., the lowest level on the feature pyramid) to detect relatively small objects. Therefore, the detection head 404 utilizes a feature pyramid to improve detection flexibility.

[0085] In one or more embodiments, the detection masking neural network 400 utilizes detection heads 404 to detect digital objects within a digital image by leveraging each detection head to generate an objectivity score for a region of the digital image. In practice, in some embodiments, while each detection head learns detections from a specific digital image dataset associated with one or more digital object categories, the detection heads generate objectivity scores for regions of the digital image during inference. However, in some cases, because each detection head learns detections based on one or more digital object categories that differ from those used by other detection heads, the objectivity scores generated by each detection head for a region of the digital image can differ. For example, in some cases, a detection head that learns on an object category will generate a higher objectivity score for a region containing digital objects from that object category compared to a detection head that learns on significantly different object categories.

[0086] Therefore, in some embodiments, the detection masking neural network 400 uses objectivity scores generated by each of the detection heads to determine whether a region of a digital image contains a digital object. For example, in some cases, the detection masking neural network 400 uses a detection threshold to determine whether the objectivity score indicates that the region contains a digital object. To illustrate, in some cases, if a threshold number of detection heads generate objectivity scores for regions that satisfy the detection threshold, the detection masking neural network 400 determines that the region contains a digital image. In some embodiments, the detection masking neural network 400 determines whether the average or combination (e.g., sum) of the objectivity scores satisfies the detection threshold. In some implementations, the detection masking neural network 400 applies weights to each objectivity score and uses weighted objectivity scores when determining whether a region contains a digital object. In some implementations, the detection masking neural network 400 uses a subset of objectivity scores from the detection heads 404 to determine whether a region contains a digital object.

[0087] As shown in the figure, a detection masking neural network 400 (e.g., a detection head 404) generates approximate boundaries 424a, 424b, 424c for digital objects 442a to 442c detected within the digital image 410. Specifically, in one or more implementations, the approximate boundaries 424a to 424c include, for example,... Figure 4C The bounding box shown.

[0088] Figure 4D The architecture of a masking head 406 of a detection masking neural network 400 according to one or more embodiments is shown. Figure 4D As shown, the masking head 406 includes multiple neural network layers. Notably, the masking head 406 includes pairs of parallel dilated convolutional layers (e.g., parallel dilated convolutional neural network layer pairs 430a to 430b). Figure 4D As shown, the convolutional layers in each pair of parallel dilated convolutional neural network layers use dilation rates of 1 and 2, respectively. The dilated convolutional layer with a dilation rate of 2 provides a receptive field size of 5, while the dilated convolutional layer with a dilation rate of 1 facilitates the localization of local information. By utilizing paired parallel dilated convolutional neural network layers, masking head 406 can analyze high-resolution features and generate high-resolution object masks if a single convolutional layer exists instead of each pair. For example, in some cases, using paired parallel dilated convolutional neural network layers allows masking head 406 to nearly double the resolution of the object mask available when using a single convolutional layer.

[0089] like Figure 4D As shown, the masking head 406 generates an object mask 432 for digital objects detected from the digital image 410. Specifically, the masking head 406 generates the object mask 432 using feature maps generated by the neural network encoder 402 for digital objects detected by the detection head 404. For example, in some cases, the masking head 406 uses approximate boundaries generated by the detection head 404 to generate an object mask for the corresponding digital object based on the feature maps.

[0090] like Figure 4D As shown, the detection masking neural network 400 generates an extended approximate boundary from the approximate boundary generated by the detection head 404 (e.g., via operation 434). Therefore, in some cases, the masking head 406 utilizes the extended approximate boundary to generate an object mask for the corresponding digit object based on the feature map. Reference will be made below. Figures 5A to 5B Let's discuss more details about the extended approximate boundary.

[0091] In one or more embodiments, the on-device masking system 106 utilizes a masking head 406 to generate at least one object mask for each digital object detected from the digital image 410. Thus, for a single digital image, in some implementations, the detection masking neural network 400 outputs multiple object masks.

[0092] As just mentioned, in one or more embodiments, the on-device masking system 106 utilizes a detection masking neural network to generate extended approximate boundaries for digital objects detected by its detection head. In some cases, the on-device masking system 106 utilizes extended approximate boundaries to avoid artifacts that occur when using closely matching approximate boundaries. Figures 5A to 5B A visual comparison of approximate boundaries and extended approximate boundaries generated for digital objects according to one or more embodiments is shown.

[0093] In particular, Figure 5A Bounding boxes 504a, 504b, and 504c generated for a digital object depicted within digital image 502 without expansion are shown. Specifically, bounding boxes 504a to 504c closely conform to the digital object, closely approaching or contacting its outer edge at one or more locations. In other words, bounding boxes 504 to 504c do not extend too far beyond the outermost edge of the digital object (if any). Figure 5A As shown, the tightness of bounding boxes 504a to 504c causes artifacts at several locations 506a, 506b, 506c, 506d, 506e, 506f, and 506g, where pixels of the detected digital objects are unnaturally glued to bounding boxes 504a to 504c. Therefore, using such bounding boxes results in inaccurate object masks that contain these artifacts.

[0094] on the contrary, Figure 5B Extended bounding boxes 510a, 510b, and 510c generated for digital objects depicted in digital image 502 are shown. As further shown, due to the extended bounding boxes 510a to 510c, artifacts caused by tight bounding boxes are not present. Therefore, in one or more embodiments, the on-device masking system 106 utilizes extended approximate boundaries to generate a more accurate object mask that does not contain such artifacts. As described above, in some cases, the on-device masking system 106 generates approximate boundaries for digital objects detected within the digital image via a detection masking neural network, and generates extended approximate boundaries from the approximate boundaries.

[0095] In some cases, the masking system 106 on the device achieves an extended approximate boundary during inference time, as referenced above. Figure 4DHowever, in some implementations, the on-device masking system 106 further utilizes an extended approximate boundary while learning the parameters of the detection masking neural network.

[0096] In one or more embodiments, on-device masking system 106 trains a detection masking neural network to detect digital objects depicted in a digital image and generate an object mask for the digital objects. Figure 6 A block diagram for training a detection masking neural network is shown according to one or more embodiments.

[0097] like Figure 6 As shown, the on-device masking system 106 trains by providing digital images from digital image datasets 602a to 602n to a detection masking neural network 604. Specifically, as shown, the detection masking neural network 604 includes a detection head 606, and the on-device masking system 106 provides digital images from digital image datasets 602a to 602n for analysis by different detection heads. For illustration, in one or more embodiments, the on-device masking system 106 provides digital images from a first digital image dataset (e.g., digital image dataset 602a) for analysis by a first detection head, provides digital images from a second digital image dataset for analysis by a second detection head, and so on.

[0098] As previously described, each digital image dataset is associated with one or more digital object categories. In other words, each digital image dataset comprises digital images associated with one or more digital object categories. Furthermore, in some embodiments, the digital object categories(s) associated with one digital image dataset are different from the digital object categories associated with other digital image datasets. Therefore, in some implementations, by associating each digital image dataset with a specific detection head, the on-device masking system 106 trains each detection head on a specific set of one or more digital object categories.

[0099] In one or more embodiments, the on-device masking system 106 provides a digital image to the detection masking neural network 604 in each training iteration. In some cases, the on-device masking system 106 provides a digital image from each of the digital image datasets 602a to 602n in each training iteration.

[0100] like Figure 6As shown, the masking system 106 on the device utilizes a detection masking neural network 604 to analyze digital images from a digital image dataset and generate predictions(s)612. Specifically, the detection masking neural network 604 utilizes a neural network encoder 610 to encode the digital image, a detection head 606 to predict digital objects depicted in the digital image, and a masking head 608 to generate a prediction object mask for the predicted digital objects. Thus, in some cases, for each iteration of training, the detection masking neural network 604 generates predictions(s)612 by generating predictions (e.g., approximate boundaries of predictions) for at least one digital object depicted in the digital image and / or generating at least one prediction object mask for the digital object.

[0101] like Figure 6 As further shown, the on-device masking system 106 compares multiple predictions 612 with multiple corresponding ground truth values ​​614 via multiple loss functions 616. For example, in one or more embodiments, the on-device masking system 106 utilizes a focal loss as the loss function in training the detection head 606. Thus, the on-device masking system 106 compares predictions (e.g., approximate boundaries of predictions) of digital objects depicted in a digital image with corresponding ground truth values ​​via the focal loss function. In some embodiments, the on-device masking system 106 utilizes other loss functions to train the detection head 606, such as cross-entropy loss. In some implementations, the on-device masking system 106 utilizes combinations of losses to train the detection head 606. Furthermore, in some embodiments, the on-device masking system 106 utilizes one or more losses or combinations of various losses (e.g., cross-entropy loss, dice loss, L1 loss, L2 loss, etc.) to train the masking head 608.

[0102] In one or more embodiments, the on-device masking system 106 backpropagates the determined loss (i.e., error) to the detection masking neural network 604 (as shown by dashed line 618) to optimize the detection masking neural network 604 by updating its parameters. In practice, the on-device masking system 106 updates parameters to minimize errors in the detection masking neural network 604, such as the errors of the detection head 606 and the masking head 608. Through multiple training iterations, the on-device masking system 106 generates a detection masking neural network with the learned network parameters 620.

[0103] In some cases, the on-device masking system 106 trains both the detection head 606 and the masking head 608 simultaneously. For example, in some cases, the on-device masking system 106 updates the parameters of one or more detection heads and masking heads 608 in each training iteration. However, in some embodiments, the on-device masking system 106 trains the detection head 606 and the masking head 608 separately.

[0104] By learning parameters for the detection head 606 using a digital image dataset associated with various digital object categories, the on-device masking system 106 flexibly detects digital objects associated with a wider range of object categories compared to conventional systems. Furthermore, the on-device masking system 106 improves the generalization of its detection model compared to conventional systems, thereby facilitating more accurate detection of digital objects not seen during training.

[0105] In some implementations, the on-device masking system 106 performs norm decoupling during training to accommodate cases where digital image datasets 602a to 602c include varying numbers of digital images. In practice, in some cases, a detection head trained on a digital image dataset with fewer digital images experiences more repetition of the same data samples compared to a detection head trained on a dataset with a larger number of digital images. Therefore, in some situations, a detection head trained on a smaller number of digital images updates its parameters more aggressively, leading to overfitting and overconfident detection scores during inference.

[0106] Therefore, in some embodiments, the on-device masking system 106 implements norm decoupling to balance the contributions of different detection heads to the detection results. For illustration, in one or more embodiments, the on-device masking system 106 decouples the length of each parameter vector (norm) from its direction (or unit vector). Furthermore, the on-device masking system 106 shares the same learnable norm among various detection heads while allowing each detection head to learn its unit vector independently.

[0107] In one or more embodiments, given D digital image datasets and W i ∈{W1,...,W D} represents the flat weights or bias parameters of the non-shared layer belonging to the i-th detector head. The masking system 106 on the device performs norm decoupling and obtains the new parameter W′. i (used to replace the original W) i ),as follows:

[0108]

[0109] In equation 3, This represents a learnable scalar norm parameter shared across all detection heads and independent of Wi. In one or more embodiments, the on-device masking system 106 maintains a consistent weight norm across the various detection heads of the detection masking neural network 604 by using norm decoupling during training.

[0110] Furthermore, as described above, in some implementations, the on-device masking system 106 utilizes Equations 1 to 2 during training to determine the global mean and global variance, and normalizes the feature maps across pyramid levels across GPUs. Therefore, the on-device masking system 106 achieves full parameter sharing across all pyramid levels of the detection head during inference.

[0111] Therefore, the masking system 106 on the device trains a detection masking neural network to detect digital objects and generate corresponding object masks. In one or more embodiments, reference is made to... Figure 6 The described algorithms and actions may include corresponding structures for performing the following steps: learning parameters for a detection masking neural network to detect digital objects depicted in digital images, utilizing predicted digital objects from multiple digital image datasets. Furthermore, refer to... Figures 4A to 4C The described detection masking neural network architecture may include a corresponding structure for performing the following steps: learning parameters of a detection masking neural network for detecting digital objects depicted in digital images, using predicted digital objects from digital images derived from multiple digital image datasets.

[0112] In some cases, the masking system 106 on the device uses upsampled masking predictions and corresponding ground truth values ​​to calculate loss during training in order to better preserve the fine details of the corresponding digital objects depicted in the digital image. Figures 7A to 7C A visual comparison of object details captured by object masks of various resolutions, according to one or more embodiments, is shown.

[0113] In particular, Figure 7A A digital image 702 depicting a digital object 708 (e.g., a plant located inside a cup) is shown. Figure 7B A low-resolution ground truth object mask 704 corresponding to the digit object 708 is shown. Such low-resolution ground truth is frequently used in conventional systems for loss calculations. However, it can be seen that the low-resolution ground truth object mask 704 fails to capture many details of the digit object 708. For example, as shown, the low-resolution ground truth object mask 704 fails to capture the details at the handle of the cup, which is part of the digit object 708. This loss of detail is typically exacerbated when the depicted digit object is large.

[0114] Therefore, by using low-resolution ground truth object masks during training, conventional systems are often unable to train their models to capture the lost details. Consequently, models trained by such systems are typically unable to generate object masks that accurately represent digital objects depicted in digital images by including these details.

[0115] Figure 7CA high-resolution ground truth object mask 706 is shown, corresponding to the digital object 708 depicted in digital image 702. As shown, the high-resolution ground truth object mask 706 captures the details of the digital object 708 better than the low-resolution ground truth object mask 704. For example, the high-resolution ground truth object mask 706 captures the details of the cup handle better.

[0116] In one or more embodiments, the on-device masking system 106 upsamples (e.g., via bilinear interpolation) the predicted object mask generated by the masking head of the detection masking neural network. The on-device masking system 106 uses the upsampled object mask and a corresponding ground truth object mask, sized to match that resolution, to perform loss calculations. Therefore, compared to systems that use a lower resolution for their ground truth values, the on-device masking system 106 trains the detection masking neural network to generate object masks that more accurately represent digital objects.

[0117] Additionally, as described above, in some cases, the on-device masking system 106 utilizes extended approximate boundaries to train the masking head of the detection masking neural network. For example, in some embodiments, the on-device masking system 106 utilizes the detection masking neural network to generate extended approximate boundaries from the predicted approximate boundaries generated by the detection head. The on-device masking system 106 further utilizes the masking head of the detection masking neural network to generate a predicted object mask based on the extended approximate boundaries. Therefore, in some cases, the on-device masking system 106 upsamples the predicted object mask generated from the extended approximate boundaries for calculating the loss.

[0118] As previously described, in some embodiments, the on-device masking system 106 generates multiple object masks from a digital image. In some cases, the on-device masking system 106 implements post-processing techniques to determine which object masks to render via a client device. Specifically, the on-device masking system 106 determines an object mask set from the generated set of object masks, which can be used for rendering via a client device. Figure 8 A block diagram is shown for determining a set of object masks for rendering using post-processing techniques, according to one or more embodiments.

[0119] like Figure 8 As shown, the on-device masking system 106 acquires an object mask 802. Specifically, the on-device masking system 106 generates the object mask 802 from a digital image using a detection masking neural network. Figure 8As further shown, the on-device masking system 106 determines a mask quality score 804 for object mask 802. For example, in some cases, the on-device masking system 106 determines a mask quality score for each object mask based on object mask 802. As used herein, the term "mask quality score" refers to a metric indicating the quality of an object mask. In particular, in some embodiments, the mask quality score refers to a value indicating the degree to which the object mask captures detail (e.g., boundary detail) of a corresponding digital object. In some embodiments, the algorithm presented below represents how the on-device masking system 106 determines the characterization of the mask quality score 804 for object mask 802.

[0120] Algorithm 1

[0121] Input: Predicted N×N mask map M

[0122] Output:Mask Quality score MaskQ

[0123] M bi =(M≥T).to(bool)

[0124] M bo : recording pixels that are on the boundaries of the masks(m ij ∈M bo =1if m ij can reach to the edge of the image without crossing anyls in M bi otherwise m ij =0

[0125] M i =(M*M bi *(1-M bo ), dim = [1, 2])

[0126] MaskQ = sum(M i ) / sum(M)

[0127] The following is a translation of the statements in Algorithm 1, for reference.

[0128] Input: The predicted N×N mask mapping M

[0129] Output: Mask quality score (MaskQ)

[0130] M bi =(M≥T).to(bool)

[0131] M bo : Record the pixels located on the boundary of the mask (if m ij It can reach the edge of the image without interacting with M. bi If any 1s intersects, then m ij ∈M bo =1, otherwise m ij =0)

[0132] M i =(M*M bi *(1-M bo ), dim = [1, 2])

[0133] MaskQ = sum(M i ) / sum(M)

[0134] In Algorithm 1, M bi This represents the binary mask obtained by comparing the generated object mask M with a threshold T. Furthermore, M... bo This represents a binary mask that records pixels located on the boundary of the object mask M. Therefore, using Algorithm 1, the on-device masking system 106 determines a mask quality score for the object mask based on statistics of the confidence scores of pixels in the object mask. Specifically, the on-device masking system 106 uses the binary mask obtained with a threshold T to calculate the average pixel-by-pixel prediction score contained in the binary mask, and uses these scores to determine the quality of the object mask.

[0135] Algorithm 1 is model-independent and has less computational overhead compared to models used by some conventional systems that use small networks to learn and predict quality scores. Therefore, by utilizing Algorithm 1, the on-device masking system 106 operates more efficiently than such systems.

[0136] Furthermore, the researchers compared the performance of an on-device masking system 106 that used Algorithm 1 to determine the mask quality score of the generated object mask. Specifically, they compared the performance of Algorithm 1 with that of a quality prediction model described by Lu Yang et al. in “Quality-aware Network for Human Parsing” (arXiv preprint arXiv:2103.05997, 2021). The researchers used Kendall’s tau coefficient to determine how well the score provided by each test model aligned with the score predicted by the learning module. Algorithm 1 performed better, achieving an alignment of 0.85, while the model described by Yang et al. achieved an alignment of 0.72. This improved performance can be attributed to Algorithm 1’s potential to address problems that arise when using other models where many interior regions are incorrectly labeled as boundaries and not included in the calculation.

[0137] like Figure 8 As further shown, the on-device masking system 106 utilizes a filtering process (e.g., represented by block 806) to determine a filter set 808 of the object mask from the object mask 802. Specifically, in one or more embodiments, the on-device masking system 106 utilizes a mask quality score 804 of the object mask 802 to determine the filter set 808 of the object mask. In some embodiments, the algorithm presented below represents a characterization of how the on-device masking system 106 determines the filter set 808 of the object mask.

[0138] Algorithm 2

[0139]

[0140] The following is a translation of the statements in Algorithm 2, for reference.

[0141] Input: n prediction tuples N = (B, M, S, M) from a detection masking neural network Q A list of (B): Bounding box; M: Object mask; S: Confidence score (objectivity score);

[0142] M Q Mask quality score

[0143] Output: Updated prediction list K = (B′, M′, S′, M′) Q )

[0144] Maintain the occupied binary mapping O, and initialize with all zeros.

[0145] Sort the n predictions from highest to lowest according to S.

[0146] For each prediction set (b) i s i ,mq i m i )

[0147] Calculate the new region: m ni =AND(m) i NOT(O))

[0148] If sum(m) ni ) / sum(m i )>T0 or s i >T1 or (mq) i >T2 and sum(m) ni ) / sum(m i )>T3)

[0149] but

[0150] (b) i s i ,mq i m i Add to K / / Add the original box and mask to the final list

[0151] Update O

[0152] otherwise

[0153] If sum(m) ni ) / sum(m i If ) > T4, then

[0154] Based on m ni Update b i

[0155] (b) i s i ,mq i m i Add to K / / Add non-overlapping boxes and masks to the final list

[0156] Update O

[0157] Finish

[0158] Finish

[0159] Finish

[0160] Therefore, using Algorithm 2, the on-device masking system 106 uses mask quality scores and confidence scores corresponding to the determined approximate boundaries to remove low-quality object masks and maintain high-quality object masks for rendering via the computing device. Specifically, in one or more embodiments, the on-device masking system 106 iterates over each object mask in the object mask set (e.g., starting with the object mask associated with the highest confidence score), uses an occupancy binary mapping to compute portions of the object mask that constitute a new region (e.g., a region not yet occupied by previously considered object masks), and determines whether to add the object mask to the filter set 808 of object masks based on whether the new region and / or some other values ​​associated with the object mask satisfy a corresponding threshold. Thus, in some embodiments, the on-device masking system 106 utilizes Algorithm 2 to filter out object masks that are low-quality due to overlap with one or more other object masks. In one or more embodiments, the thresholds used in Algorithms 1 through 2 are configurable.

[0161] By determining the filter set 808 of the object mask, the on-device masking system 106 facilitates the rendering of high-quality object masks via a computing device. Specifically, in some embodiments, the on-device masking system 106 facilitates the provision of object masks that do not overlap with each other. Furthermore, in some cases, the on-device masking system 106 removes false positives that overlap with the high-quality mask while selecting smaller, more reliable areas.

[0162] Researchers compared the performance of on-device masking system 106 using post-processing techniques characterized by Algorithms 1 to 2 with that of on-device masking system 106 without post-processing. Using a 50% joint union (IoU), on-device masking system 106 improved the recall of the obtained object mask from 66.2% to 67.4% using post-processing techniques.

[0163] Turn Figure 9 Additional details regarding the various components and capabilities of the shielding system 106 on the device will now be provided. Specifically, Figure 9 The diagram shows a computing device 900 (e.g., referenced above). Figure 1 The on-device masking system 106 is implemented in one of the discussed server(s) 102 and / or one of the client devices 110a to 110n. Furthermore, the on-device masking system 106 is also part of the image editing system 104. As shown, in one or more embodiments, the on-device masking system 106 includes, but is not limited to, a neural network training manager 902, a neural network application engine 904, a post-processing engine 906, a user interface manager 908, and a data storage 910 (which includes a detection masking neural network 912 and a digital image dataset 914).

[0164] As just mentioned, and as Figure 9 As shown, the on-device masking system 106 includes a neural network training manager 902. In one or more embodiments, the on-device masking system 106 trains a detection masking neural network to detect digital objects within a digital image and generate an object mask for the detected digital objects. For example, in some embodiments, the neural network training manager 902 trains multiple detection heads of the detection masking neural network to detect digital objects using different digital image datasets associated with different categories of digital objects. Furthermore, the on-device masking system 106 trains masking heads of the detection masking neural network to generate an object mask for the digital objects identified by the detection heads.

[0165] In addition, such as Figure 9 As shown, the masking system 106 on the device includes a neural network application engine 904. In one or more embodiments, the neural network application engine 904 implements a detection masking neural network trained by a neural network training manager 902. For example, in some cases, the neural network application engine 904 receives a digital image and utilizes the detection masking neural network to generate an object mask for a digital object depicted in the digital image.

[0166] like Figure 9 As shown, the on-device masking system 106 also includes a post-processing engine 906. In one or more embodiments, the post-processing engine 906 determines a set of object masks that can be used for rendering via a computing device. For example, in some cases, the post-processing engine 906 determines a mask quality score for an object mask generated by a detection masking neural network. The post-processing engine 906 further utilizes the mask quality score and / or other attributes of the generated object masks to filter out low-quality masks and determine a filtered set of object masks for rendering.

[0167] Furthermore, as shown in the figure, the masking system 106 on the device includes a user interface manager 908. In one or more embodiments, the user interface manager 908 manages the rendering of one or more object masks via a computing device (e.g., via a graphical user interface of computing device 900). For example, in some cases, the user interface manager 908 detects a selection of a digital object depicted within a digital image. In response to detecting this selection, the user interface manager 908 retrieves the corresponding object mask generated by the detection masking neural network and provides the object mask for display.

[0168] like Figure 9 As shown, the masking system 106 on the device also includes a data storage 910. Specifically, the data storage 910 includes a detection masking neural network 912 and a digital image dataset 914.

[0169] Each of components 902 to 914 of the masking system 106 on the device may include software, hardware, or both. For example, components 902 to 914 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as a client device or a server device. When executed by one or more processors, the computer-executable instructions of the masking system 106 on the device may cause the computing device(s) to perform the methods described herein. Alternatively, components 902 to 914 may include hardware, such as a dedicated processing device for performing a particular function or group of functions. Alternatively, components 902 to 914 of the masking system 106 on the device may include a combination of computer-executable instructions and hardware.

[0170] Furthermore, components 902 to 914 of the on-device masking system 106 can be implemented, for example, as one or more operating systems, one or more standalone applications, one or more modules of an application, one or more plugins, one or more library functions that can be called by other applications, and / or a cloud computing model. Therefore, components 902 to 914 of the on-device masking system 106 can be implemented as standalone applications, such as desktop or mobile applications. Additionally, components 902 to 914 of the on-device masking system 106 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, components 902 to 914 of the on-device masking system 106 can be implemented in mobile device applications or "app" suites. For example, in one or more embodiments, the on-device masking system 106 may include or operate in conjunction with digital software applications, such as... or “ADOBE”, “PHOTOSHOP” and “LIGHTROOM” are registered trademarks or trademarks of Adobe Inc. in the U.S. and / or other countries.

[0171] refer to Figures 1 to 9 The corresponding text and examples provide many different methods, systems, devices, and non-transitory computer-readable media for the masking system 106 on the device. In addition to the foregoing, one or more embodiments may also be described according to flowcharts including actions for achieving a particular result, such as... Figure 10 As shown. Figure 10 It can be performed with more or fewer actions. Furthermore, these actions can be performed in different orders. Additionally, the actions described herein can be repeated or performed in parallel with each other, or performed in parallel with different instances of the same or similar actions.

[0172] Figure 10A flowchart illustrating a series of actions 1000 for generating a digital image of a digital object using a detection masking neural network, according to one or more embodiments, is shown. Although Figure 10 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 10 Any action shown. In some implementations, Figure 10 The action is performed as part of the method. For example, in some embodiments, Figure 10 The actions are performed within a digital media environment used for editing digital images as part of a computer implementation method for constructing or implementing a neural network for image segmentation. Alternatively, a non-transitory computer-readable medium may store instructions thereon that, when executed by at least one processor, cause the computing device to perform... Figure 10 In some embodiments, the system performs the following actions. Figure 10 The system includes, for example, at least one memory device in one or more embodiments, a detection masking neural network comprising: a neural network encoder for extracting encoded feature maps from a digital image; a plurality of detection heads, each including at least one neural network layer and corresponding to one or more digital object categories, the digital object categories being different from those corresponding to other detection heads among the plurality of detection heads; and a masking head including additional neural network layers. The system also includes components configured to cause the system to perform... Figure 10 At least one server device for the action.

[0173] A series of actions 1000 includes action 1002 of receiving a digital image. For example, in one or more embodiments, action 1002 relates to receiving a digital image at a computing device. In some cases, the digital image depicts a single digital object. However, in some implementations, the digital image depicts multiple digital objects.

[0174] The series of actions 1000 also includes action 1004 of detecting digital objects from a digital image using a neural network detection head. For example, in some embodiments, action 1004 involves detecting digital objects depicted in a digital image at a computing device using a detection head of a detection masking neural network. In one or more embodiments, the on-device masking system 106 detects digital objects depicted in a digital image at a computing device using a detection head of a detection masking neural network by utilizing a detection head having multi-scale synchronous batch normalization neural network layers.

[0175] In some implementations, the on-device masking system 106 detects digital objects depicted in a digital image at a computing device using a detection head of a detection masking neural network: generating an objectivity score for a portion of the digital image using the detection head at the computing device; and using the objectivity score to determine that the portion of the digital image corresponds to a digital object. In some embodiments, the on-device masking system 106 generates an additional objectivity score for the portion of the digital image at a computing device using an additional detection head of the detection masking neural network. Therefore, in some cases, determining that the portion of the digital image corresponds to a digital object using the objectivity score includes: determining that the portion of the digital image corresponds to a digital object using both the objectivity score and the additional objectivity score. In at least one implementation, determining that the portion of the digital image corresponds to a digital object using both the objectivity score and the additional objectivity score includes: determining that the portion of the digital image corresponds to a digital object using weights for the objectivity score and additional weights for the additional objectivity score, the weights and additional weights being determined via norm decoupling of the detection head and the additional detection head.

[0176] In one or more embodiments, the on-device masking system 106 utilizes a neural network encoder of a detection masking neural network at a computing device to generate a set of feature maps corresponding to multiple levels of image features from a digital image. Thus, in some cases, by utilizing a detection head based on this feature map at a computing device to detect digital objects, the on-device masking system 106 utilizes a detection head of a detection masking neural network at a computing device to detect digital objects depicted in a digital image.

[0177] In one or more embodiments, the on-device masking system 106 detects digital objects depicted in a digital image by using a detection head at a computing device to determine approximate boundaries corresponding to digital objects depicted in the digital image. In some cases, the on-device masking system 106 also generates extended approximate boundaries for the digital objects based on the approximate boundaries.

[0178] The series of actions 1000 also includes action 1006 of generating an object mask for a digital object using a neural network masking head. For example, in some cases, action 1006 involves generating an object mask for a digital object at a computing device using a masking head of a detection masking neural network.

[0179] In one or more embodiments, an on-device masking system 106 generates an object mask for a digital object by utilizing a masking head of a detection masking neural network at a computing device based on an extended approximate boundary for the digital object.

[0180] In one or more embodiments, the series of actions 1000 further includes actions for performing one or more post-processing techniques after generating one or more object masks from a digital image. For example, in some cases, these actions include: determining confidence scores for pixels of the object mask generated for the digital object; generating a binary mask corresponding to the object mask using the confidence scores for the pixels; and generating a mask quality score for the object mask using the binary mask and the confidence scores for the pixels. In some cases, the on-device masking system 106 uses at least one of the following to determine whether to include the object mask generated for the digital object in a set of object masks: the object mask, a bounding box corresponding to the digital object, a confidence score corresponding to the bounding box, and a mask quality score for the object mask; and provides an object mask for display on a computing device based on including the object mask in the set of object masks.

[0181] As previously described, in one or more embodiments, a series of actions 1000 includes actions for constructing a neural network for image segmentation. For example, in some cases, these actions include: accessing multiple digital image datasets, the multiple digital image datasets comprising digital images depicting at least one digital object with associated category labels, wherein each digital image dataset from the multiple digital image datasets corresponds to one or more digital object categories, the one or more digital object categories being different from digital object categories corresponding to other digital image datasets from the multiple digital image datasets; and utilizing a detection masking neural network to predict digital objects depicted in the digital images of the multiple digital image datasets, the detection masking neural network including a detection head for each digital image dataset, the detection head being configured to detect digital objects from one or more digital object categories of the digital image datasets. In some implementations, these actions further include: using the predicted digital objects to learn parameters for the detection masking neural network. The on-device masking system 106 may further implement the detection masking neural network by utilizing the detection masking neural network with the learned parameters to generate one or more objectivity scores for digital objects to detect digital objects depicted in digital images. In some cases, the masking system 106 on the device also utilizes the masking head of the detection masking neural network to generate an object mask for digital objects detected from a digital image.

[0182] For illustration, in one or more embodiments, on-device masking system 106 generates multiple coded feature maps from a digital image using a neural network encoder; detects digital objects depicted in the digital image using multiple detection heads based on the multiple coded feature maps; and generates an object mask for the digital objects using a masking head. In some cases, on-device masking system 106 detects digital objects depicted in a digital image using multiple detection heads based on the multiple coded feature maps by: generating multiple class-agnostic object scores for a portion of the digital image using the multiple detection heads based on the multiple coded feature maps; and determining that the portion of the digital image corresponds to a digital object using the multiple class-agnostic object scores. In some cases, determining that the portion of the digital image corresponds to a digital object using the multiple class-agnostic object scores includes: determining that the portion of the digital image corresponds to a digital object based on determining that one or more object scores from the multiple class-agnostic object scores satisfy a detection threshold.

[0183] In one or more embodiments, the on-device masking system 106 further generates an additional object mask for one or more digital objects depicted in a digital image; generates an occupancy binary map corresponding to the digital image; determines, by comparing the object mask and the additional object mask with the occupancy binary map, to include at least one of the object masks or additional object masks in an object mask set; and provides one or more digital object masks for display on a client device based on including one or more digital object masks in the object mask set.

[0184] In one or more embodiments, multiple detection heads of a detection masking neural network share a set of convolutional parameters, and each detection head includes a set of output parameters learned separately from the output parameters of other detection heads. In some embodiments, additional neural network layers of the masking heads of the detection masking neural network include parallel dilated convolutional neural network layers that capture local features associated with digital objects. Furthermore, in some cases, the multiple detection heads include multi-scale synchronous batch normalization neural network layers that normalize multiple encoded feature maps generated from digital images.

[0185] Embodiments of this disclosure may include or utilize a dedicated or general-purpose computer including computer hardware (e.g., one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory) and executes those instructions to perform one or more processes, including one or more processes described herein.

[0186] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0187] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0188] "Network" is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer correctly regards that connection as a transmission medium. A transmission medium may include networks and / or data links that can be used to carry desired program code in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or special-purpose computer. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0189] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-transitory computer-readable storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or to a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that a non-transitory computer-readable storage medium (device) can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0190] Computer-executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. In some embodiments, the computer-executable instructions execute on a general-purpose computer to turn the general-purpose computer into a special-purpose computer that implements the elements of this disclosure. The computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0191] Those skilled in the art will understand that this disclosure can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. This disclosure can also be practiced in distributed system environments, where local and remote computer systems linked via a network (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links) each perform tasks. In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0192] The embodiments of this disclosure can also be implemented in a cloud computing environment. In this description, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly configured via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly.

[0193] Cloud computing models can be composed of various features, such as on-demand self-service, multiple network access, resource pooling, rapid elasticity, and measurable services. Cloud computing models can also expose various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Different deployment models (such as private cloud, community cloud, public cloud, hybrid cloud, etc.) can also be used to deploy cloud computing models. In this specification and claims, a “cloud computing environment” means an environment in which cloud computing is employed.

[0194] Figure 11 A block diagram of an example computing device 1100 that can be configured to perform one or more of the processes described above is shown. It should be understood that one or more computing devices, such as computing device 1100, can represent the aforementioned computing devices (e.g., servers 102 and / or client devices 110a to 110n). In one or more embodiments, computing device 1100 may be a mobile device (e.g., mobile phone, smartphone, PDA, tablet computer, laptop computer, camera, tracker, watch, wearable device). In some embodiments, computing device 1100 may be a non-mobile device (e.g., desktop computer or another type of client device). Furthermore, computing device 1100 may be a server device including cloud-based processing and storage capabilities.

[0195] like Figure 11 As shown, computing device 1100 may include one or more processors 1102, memory 1104, storage device 1106, input / output interface 1108 (or "I / O interface 1108"), and communication interface 1110 that can be communicatively coupled via a communication infrastructure (e.g., bus 1112). Although in Figure 11 The image shows a computing device 1100, but... Figure 11 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in some embodiments, the computing device 1100 includes fewer than [number of components]. Figure 11 The components shown. A more detailed description will now follow. Figure 11 The components of the computing device 1100 shown.

[0196] In a particular embodiment, processor(s) 1102 includes hardware for executing instructions such as instructions constituting a computer program. By way of example and not limitation, in order to execute instructions, processor(s) 1102 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1104, or storage device 1106, and decode and execute the instructions.

[0197] Computing device 1100 includes memory 1104 coupled to processor(s) 1102. Memory 1104 can be used to store data, metadata, and programs executed by processor(s). Memory 1104 can include one or more types of memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage. Memory 1104 can be internal or distributed memory.

[0198] Computing device 1100 includes storage device 1106, which includes storage means for storing data or instructions. By way of example and not limitation, storage device 1106 may include the aforementioned non-transitory storage media. Storage device 1106 may include hard disk drive (HDD), flash memory, universal serial bus (USB) drive, or combinations of these or other storage devices.

[0199] As shown in the figure, computing device 1100 includes one or more I / O interfaces 1108 provided to allow a user to provide input (e.g., user strokes) to computing device 1100, receive output from computing device 1100, and otherwise transmit data to and from computing device 1100. These I / O interfaces 1108 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O interfaces 1108. Touchscreens can be activated with a stylus or finger.

[0200] I / O interface 1108 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O interface 1108 is configured to provide graphics data to the display for presentation to a user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0201] The computing device 1100 may also include a communication interface 1110. The communication interface 1110 may include hardware, software, or both. The communication interface 1110 provides one or more interfaces for communication (e.g., packet-based communication) between the computing device and one or more other computing devices or one or more networks. By way of example and not limitation, the communication interface 1110 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 1100 may also include a bus 1112. The bus 1112 may include hardware, software, or both for connecting components of the computing device 1100 to each other.

[0202] In the foregoing description, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention have been described with reference to the details discussed herein, and various embodiments are illustrated in the accompanying drawings. The above description and drawings are illustrative of the invention and should not be construed as limiting the invention. Numerous specific details have been described to provide a thorough understanding of various embodiments of the invention.

[0203] The invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered illustrative rather than restrictive in all respects. For example, the methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. Furthermore, the steps / actions described herein may be repeated, or performed in parallel with each other or in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All modifications falling within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A computer-implemented method for constructing a neural network for image segmentation in a digital media environment used for editing digital images, comprising: Access multiple digital image datasets, the multiple digital image datasets comprising digital images depicting at least one digital object with associated category labels, wherein each digital image dataset from the multiple digital image datasets corresponds to one or more digital object categories, the one or more digital object categories being different from digital object categories corresponding to other digital image datasets from the multiple digital image datasets; A detection masking neural network is used to predict digital objects depicted in digital images of the plurality of digital image datasets. The detection masking neural network includes a detection head for each digital image dataset, the detection head being configured to detect digital objects from one or more digital object categories of the digital image dataset. as well as The following steps are performed: using the predicted digital objects from the digital images from the plurality of digital image datasets, parameters for a detection masking neural network for detecting digital objects depicted in the digital images are learned.

2. The computer-implemented method according to claim 1 further includes: Digital objects depicted in a digital image are detected by generating one or more objectivity scores for the digital objects using the detection masking neural network with the learned parameters.

3. The computer-implemented method according to claim 2 further includes: The masking head of the detection masking neural network is used to generate an object mask for the digital object detected from the digital image.

4. A non-transitory computer-readable medium storing instructions thereon, the instructions causing a computing device to perform operations when executed by at least one processor, the operations including: Receive digital images at the computing device; At the computing device, a plurality of detection heads of a detection masking neural network are used to detect digital objects depicted in the digital image, wherein each of the plurality of detection heads corresponds to a digital object category, which is different from the digital object categories corresponding to the other detection heads of the plurality of detection heads. as well as An object mask for the digital object is generated at the computing device using the masking head of the detection masking neural network.

5. The non-transitory computer-readable medium of claim 4, further comprising instructions that, when executed by the at least one processor, cause the computing device to perform additional operations, said additional operations including: At the computing device, a neural network encoder of the detection masking neural network is used to generate a set of feature maps corresponding to image features from multiple levels of the digital image; as well as Detecting the digital object depicted in the digital image using the plurality of detection heads of the detection masking neural network at the computing device includes: detecting the digital object at the computing device using the plurality of detection heads based on the feature map set.

6. The non-transitory computer-readable medium of claim 4, wherein detecting the digital object depicted in the digital image using the plurality of detection heads of the detection masking neural network at the computing device comprises: The digital object is detected using the detection head with a multi-scale synchronous batch normalized neural network layer.

7. The non-transitory computer-readable medium of claim 4, wherein detecting the digital object depicted in the digital image using the plurality of detection heads of the detection masking neural network at the computing device comprises: At the computing device, an objectivity score for a portion of the digital image is generated using one of the plurality of detection heads. as well as The objectivity score is used to determine whether the portion of the digital image corresponds to the digital object.

8. The non-transitory computer-readable medium of claim 7, further comprising instructions that, when executed by the at least one processor, cause the computing device to perform additional operations, said additional operations including: Additional objectivity scores for the portions of the digital image are generated at the computing device using additional detection heads from the plurality of detection heads. The determination of whether a portion of a digital image corresponds to a digital object using the objectivity score includes: determining whether a portion of a digital image corresponds to a digital object using the objectivity score and the additional objectivity score.

9. The non-transitory computer-readable medium of claim 8, wherein determining that the portion of the digital image corresponds to the digital object using the objectivity score and the additional objectivity score comprises: The portion of the digital image corresponding to the digital object is determined by using weights for the objectivity score and additional weights for the additional objectivity score, the weights and the additional weights being determined via norm decoupling of the detection head and the additional detection head.

10. The non-transitory computer-readable medium according to claim 4, The detection of the digital objects depicted in the digital image using the plurality of detection heads of the detection masking neural network at the computing device includes: At the computing device, one of the multiple detection heads is used to determine an approximate boundary corresponding to the digital object depicted in the digital image; and It also includes instructions that, when executed by the at least one processor, cause the computing device to perform additional operations, the additional operations including: generating an extended approximate boundary for the digital object based on the approximate boundary.

11. The non-transitory computer-readable medium of claim 10, wherein generating the object mask for the digital object at the computing device using the mask head of the detection masking neural network comprises: At the computing device, the masking head of the detection masking neural network is used to generate the object mask based on the extended approximate boundary for the digital object.

12. The non-transitory computer-readable medium of claim 4, further comprising instructions that, when executed by the at least one processor, cause the computing device to perform additional operations, said additional operations including: Determine the confidence score of the pixels of the object mask generated for the digital object; A binary mask corresponding to the object mask is generated using the confidence score of the pixel. as well as A mask quality score for the object mask is generated using the binary mask and the confidence score for the pixel.

13. The non-transitory computer-readable medium of claim 12, further comprising instructions, when executed by the at least one processor, causing the computing device to perform the following additional operations, said additional operations including: The object mask generated for the digital object is included in the object mask set using at least one of the following: the object mask, the bounding box corresponding to the digital object, the confidence score corresponding to the bounding box, and the mask quality score for the object mask; as well as The object mask is provided for display on the computing device by including the object mask in the object mask set.

14. A system comprising: At least one memory device, including a detection masking neural network, said detection masking neural network comprising: Neural encoders extract encoded feature maps from digital images; Multiple detection heads, each including at least one neural network layer and corresponding to one or more digital object categories, said one or more digital object categories being different from digital object categories corresponding to other detection heads from the plurality of detection heads; and Masking head, including additional neural network layers; and At least one server device is configured to enable the system to: The neural network encoder is used to generate multiple coded feature maps from a digital image; The plurality of detection heads are used to detect digital objects depicted in the digital image based on the plurality of coded feature maps; and The masking head is used to generate an object mask for the digital object.

15. The system of claim 14, wherein the at least one server device is configured to enable the system to detect the digital object depicted in the digital image based on the plurality of coded feature maps using the plurality of detection heads in such a way that: The plurality of detection heads are used to generate multiple class-agnostic object identity scores for portions of the digital image based on the plurality of coded feature maps; and The digital image portion is determined to correspond to the digital object using the multiple category-agnostic objectness scores.

16. The system of claim 15, wherein determining that the portion of the digital image corresponds to the digital object using the plurality of category-agnostic objectivity scores comprises: The portion of the digital image is determined to correspond to the digital object based on determining that one or more object scores from the plurality of category-agnostic object scores satisfy a detection threshold.

17. The system of claim 14, wherein the at least one server device is further configured to cause the system to: Generate an additional object mask for one or more digital objects depicted in the digital image; Generate an occupancy binary map corresponding to the digital image; The object mask and the additional object mask are compared with the occupancy binary map to determine whether to include at least one of the additional object masks in the object mask set; as well as One or more digital object masks are provided for display on a client device by including the one or more digital object masks in the object mask set.

18. The system of claim 14, wherein the additional neural network layer of the masking head comprises a parallel dilated convolutional neural network layer that captures local features associated with digital objects.

19. The system of claim 14, wherein the plurality of detection heads share a set of convolutional parameters, and each detection head includes a set of output parameters learned separately from the output parameters of the other detection heads.

20. The system of claim 14, wherein the plurality of detection heads includes a multi-scale synchronous batch normalization neural network layer, the multi-scale synchronous batch normalization neural network layer normalizing the plurality of coded feature maps generated from the digital image.

Images