Leverage object attribute detection models to automatically select instances of detected objects in images

Abstract

The present disclosure relates to an object selection system that accurately detects and automatically selects target instances of user-requested objects (e.g., query object instances) in digital images. In one or more embodiments, the object selection system can analyze one or more user inputs to determine an optimal object attribute detection model from a plurality of specialized and general object attribute models. Additionally, the object selection system can utilize the selected object attribute model to detect and select one or more target instances of a query object in an image, wherein the image includes multiple instances of the query object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to concurrently filed U.S. patent application serial number 16 / 518,795 (attorney docket number 20030.257.4) filed on July 22, 2019 and U.S. patent application serial number 16 / 518,810 (attorney docket number 20030.257.2) filed on July 22, 2019, both of which are incorporated herein by reference in their entirety. Background Art

[0003] In recent years, digital image editing has greatly increased. In fact, the advancement of both hardware and software has improved the ability of individuals to capture, create and edit digital images. For example, the hardware on most modern computing devices (such as servers, desktops, notebook computers, tablet computers and smart phones) can realize digital image editing without obvious delay time or processing delay. Similarly, the improvement of software enables individuals to modify, combine, filter or otherwise edit digital images. The example of editing digital images includes detecting objects, copying objects from an image to a new background, or deleting objects from an image.

[0004] Despite these improvements in digital image editing, conventional systems have many problems with flexibility, accuracy, and operating efficiency in image editing systems, particularly in detecting and selecting objects in digital images. For example, many conventional systems are limited in the types of objects they can detect in an image. In fact, conventional image editing systems are often limited to a small number of object categories. In addition, conventional systems primarily detect only a small number of objects within an object category.

[0005] Additionally, conventional systems are inaccurate. For example, many conventional systems that provide object detection often identify the wrong object or fail to identify any object. For example, when a user desires a selection of a target instance of an object, conventional systems often identify and group multiple instances of the object in a single selection because these systems cannot distinguish between instances of the object being detected in the image.

[0006] In addition, some conventional systems are inaccurate because they provide imprecise results. For example, although some conventional systems can select individual masks of objects, the boundaries of these masks are rough and imprecise. In fact, these conventional systems often produce object masks that include additional parts of the image above or do not include the entire object. As a result, the user must manually correct the selection and / or manually select the desired object.

[0007] Furthermore, conventional systems are inefficient. For example, conventional systems have significant deficiencies in automatic object detection and selection. For example, many conventional systems for detecting objects are end-to-end neural networks. When something fails or fails to achieve the desired results, the user of the system, or even the creator, has no way to find out which component of the system is not working properly. Instead, the entire system must be retrained and tweaked until the desired results are achieved, a process that can take a lot of time and computing resources.

[0008] In addition, many conventional systems provide inefficient mouse-based input tools that further require the user to manually select the desired object. These tools are often imprecise and challenging for many selection tasks. As a result, users spend a lot of time and the user's interaction with different selection tools wastes a lot of computing resources on detecting, displaying, selecting and correcting object selections in digital images.

[0009] These, as well as other problems and issues exist in image editing systems regarding detecting and selecting objects in digital images. Summary of the invention

[0010] Embodiments of the present disclosure utilize systems, non-transitory computer-readable media, and methods for automatically selecting a target instance of an object in a digital image based on a natural language-based input to provide benefits and / or solve one or more of the foregoing or other problems in the art. For example, the disclosed system can analyze one or more user inputs to determine an optimal object attribute detection model (e.g., an object attribute neural network) from a plurality of specialized and general object attribute models. In addition, the disclosed system can utilize the selected object attribute model to identify a target instance of an object in an image that includes multiple instances of the object.

[0011] To illustrate, the disclosed system may identify a user selection query requesting selection of a target instance of an object within an image. Specifically, the user selection query may indicate a query object and object attributes corresponding to the query object. Based on the user selection query, the disclosed system may detect multiple instances of the query object in the image. As an example, if there are multiple instances of a desired object in the image, the disclosed system may identify and select each instance individually. Additionally, the disclosed system may determine an object attribute detection model based on the object attributes and utilize the object attribute detection model to select a specific object instance. Furthermore, the disclosed system may provide a selected specific object instance selected in the image.

[0012] Indeed, in many embodiments, the disclosed system can accurately detect specified instances of an object based on a user query. For example, when there are multiple instances of a desired object in an image, the disclosed system can identify and select each instance individually. Additionally, the disclosed system can identify specified attributes from a selection query and select instances of an object that match the specified attributes. In various embodiments, the disclosed system can select instances of an object based on color, positioning, size, location, material, expression, shape, or other attributes.

[0013] Although the present disclosure refers to systems for simplicity, the present disclosure also applies to certain disclosed methods and non-transient computer-readable media. The following description sets forth additional features and advantages of one or more embodiments of the disclosed systems, computer media, and methods. In some cases, such features and advantages will be apparent to those skilled in the art from the description or may be learned through the disclosed practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] As briefly described below, the detailed description provides additional specificity and detail for one or more embodiments through use of the accompanying figures.

[0015] Figure 1 A schematic diagram of an environment in which an object selection system operates in accordance with one or more embodiments is shown.

[0016] Figure 2 A schematic diagram of automatically detecting and selecting a target query object instance in an image according to one or more embodiments is shown.

[0017] FIG. 3A to FIG. 3B A graphical user interface workflow is shown for automatically detecting and selecting a query object in an image in accordance with one or more embodiments.

[0018] Figure 4 A schematic diagram of an object selection pipeline for selecting a target query object in an image is shown in accordance with one or more embodiments.

[0019] Figure 5 A flow diagram of analyzing a user selection query to identify query objects is shown in accordance with one or more embodiments.

[0020] Figure 6 A flow chart is shown for detecting multiple query object instances in an image using an object detection network in accordance with one or more embodiments.

[0021] FIG. 7A to FIG. 7C A graphical user interface workflow is shown for utilizing an object detection network to detect multiple query object instances in an image in accordance with one or more embodiments.

[0022] Figure 8 A flow chart for detecting a target instance of a query object is shown in accordance with one or more embodiments.

[0023] 9A to 9D A graphical user interface workflow is shown for detecting target instances of a query object using an object property detection network in accordance with one or more embodiments.

[0024] Fig.10 A flow chart is shown for detecting target instances of a query object using an object color attribute detection neural network in accordance with one or more embodiments.

[0025] Fig.11 A graphical user interface workflow is shown for selecting query object instances based on object color attributes in accordance with one or more embodiments.

[0026] Fig.12 A flow chart of using an object localization attribute model to detect target instances of a query object according to one or more embodiments is shown.

[0027] Fig.13 A graphical user interface workflow for selecting query object instances based on object location attributes in accordance with one or more embodiments is shown.

[0028] Fig.14 A flow chart is shown for detecting target instances of a query object using a generic object attribute detection neural network in accordance with one or more embodiments.

[0029] Fig.15 A flow diagram is shown for detecting and selecting target instances of a query object in an image based on multimodal input in accordance with one or more embodiments.

[0030] Fig.16 A schematic diagram of an object selection system is shown in accordance with one or more embodiments.

[0031] Fig.17 A flow chart illustrating a series of actions for detecting a query object instance using one of a plurality of object property detection models according to one or more embodiments.

[0032] Fig.18 A block diagram of an example computing device for implementing one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The present disclosure describes one or more embodiments of an object selection system that accurately detects and automatically selects target instances of a user-requested object (e.g., a query object instance) in a digital image. Specifically, in one or more embodiments, the object selection system can analyze one or more user inputs to determine an optimal object attribute detection model from a plurality of specialized and general object attribute models. Additionally, the object selection system can utilize the selected object attribute model to detect and select one or more target instances of a query object in an image, wherein the image includes multiple instances of the query object.

[0034] To illustrate, the object selection system may identify a user selection query (e.g., a text string) requesting selection of a particular object instance within an image. Specifically, the selection query may indicate a query object and object attributes corresponding to the query object. Based on the selection query, the object selection system may detect multiple instances of the query object in the image. Additionally, the object selection system may determine which object attribute detection model to utilize based on analyzing the object attributes in the selection query. Furthermore, the object selection system may utilize the determined object attribute detection model to detect and then select a target query object instance in the image. Finally, the object selection system may automatically select the detected query object instance for the user within the image.

[0035] In one or more embodiments, the object selection system can detect objects corresponding to a variety of object types and categories. In this manner, a user can request the object selection system to find a large number of objects. In various embodiments, the object selection system can utilize an object selection pipeline that provides a framework for the object selection system to determine which object detection neural networks to utilize to detect a target query object instance requested by the user.

[0036] To illustrate, in one or more embodiments, the object selection system can detect a selection query (e.g., a text string) from a user that requests automatic selection of target instances of an object within an image and / or editing of the image for a particular object instance. Additionally, the object selection system can parse the query text string (hereinafter referred to as the "query string") and / or utilize natural language processing to determine various parts of speech, such as nouns and adjectives within the query. For example, the nouns in the query string correspond to the query objects that the user is requesting the object selection system to automatically select in the image. Furthermore, the adjectives in the query string identify which instances of the query object are to be selected when multiple query objects are included in the image.

[0037] In one or more embodiments, the object selection system can analyze the query object in the query string to determine the object category. Based on the object category, the object selection system can select and utilize an object detection neural network to detect the query object within the image. In additional embodiments, the object selection system can detect multiple instances of the query object within the image.

[0038] In various embodiments, upon detecting a query object, the object selection system generates an approximate boundary (e.g., a bounding box) around the object within the image. In additional embodiments, the object selection system provides the detected query object to an object mask neural network. In various embodiments, the object mask neural network generates an object mask for the detected object. In effect, the object mask enables the object selection system to select the query object detected in the image.

[0039] In some embodiments, the object selection system generates multiple bounding boxes and / or object masks for multiple instances of the detected query object. In these embodiments, the object selection system can provide the multiple instances of the detected query object to the user. In additional embodiments, when multiple instances of the detected query object are detected, the object selection system can provide an individual selection of each instance.

[0040] As described above, in some embodiments, the query string includes one or more object attributes of a target instance of a query object. For example, the query string indicates size, color, location, position, material, emotion, or another attribute of the query object. In response to detecting the object attribute in the query string, the object selection system can further determine which instance of the query object to select. Specifically, the object selection system can utilize an object attribute detection model (e.g., a neural network or a metric) to determine which instance of the detected query object to select and provide it to the user in the image.

[0041] In one or more embodiments, the object selection system detects that the object attribute corresponds to a color. In these embodiments, the object selection system may select and utilize an object color detection model to identify instances of the detected object query that match the color specified in the query string. In some embodiments, the object selection system detects that the object attribute corresponds to a location. In these embodiments, the object selection system may select and utilize a location model to identify instances of the detected object query that match the specified location. In still other embodiments, the object selection system may select and utilize another object attribute detection neural network to select the target instance specified in the query string.

[0042] In various embodiments, the object selection system facilitates multi-modal user input to identify a target instance of a query object. For example, the object selection system utilizes both a query string and a mouse / touch input to identify a target instance of a query object. In some embodiments, the mouse / touch input provides an indication or selection of a specific object instance requested by the user. In some embodiments, the object selection system can select an object instance in an image based solely on the user's mouse / touch input (e.g., without "verbal" user input).

[0043] As previously described, object selection systems have many advantages, benefits, and practical applications compared to conventional systems. For example, and as described above, object selection systems perform a variety of functions with respect to object detection, isolation, and selection. In practice, the object selection system can detect a specific object instance in an image requested by a user and generate an object mask that selects the specific object instance within the image. In practice, the object selection system can determine that the user is requesting a target instance of an object within the image and provide the specific requested instance. Thus, object selection systems provide increased flexibility and expanded functionality compared to conventional systems.

[0044] In addition, the object selection system provides greater flexibility by detecting target instances of objects in an image based on multiple user inputs. For example, the object selection system utilizes both a query string (e.g., a "language" prompt) and a mouse or touch input selection to detect the requested object instance. In fact, as described above, the object selection system can also select the target instance of the requested object based on object attributes detected in a selection query (e.g., a query string).

[0045] As another example, the object selection system provides higher accuracy than conventional systems. For example, the object selection system improves object detection accuracy by determining an object attribute detection model that best corresponds to a query object attribute from a set of different object attribute detection models. In fact, by utilizing an object attribute detection model or neural network tailored to the query object, the object selection system achieves improved accuracy of object detection and object selection.

[0046] In addition, the object selection system provides improved accuracy over conventional systems by isolating selections of multiple instances of a detected object rather than providing a single selection that includes all instances of the detected object. In addition, the object selection system can detect a user requesting a target instance (e.g., by detecting object attributes in a query string), isolate the target instance using multiple object detection models, and provide a selection of the target instance of the object requested by the user.

[0047] Furthermore, by utilizing an object selection pipeline, the object selection system provides improved efficiency over conventional systems. Indeed, unlike closed end-to-end conventional systems, when a failure occurs, the object selection system can pinpoint the failed component in the object selection pipeline and repair the component. Furthermore, the object selection system can add additional components to improve accuracy. For example, the object selection system can add additional specialized object detection neural networks to the object selection pipeline corresponding to frequently queried objects. Similarly, the object selection system can replace and / or upgrade components within the object selection pipeline with more efficient versions.

[0048] In addition, the object selection system greatly reduces the number of actions that many conventional systems require the user to perform to select an object within an image. Rather than manually selecting an object using an inefficient mouse-based input tool, the user "tells" the object selection system (e.g., by providing a verbal prompt in a query string) which object to select, and the object detection system automatically detects and selects the object. In effect, the object selection system greatly simplifies the object selection process for the user, reducing it to one or two simple actions.

[0049] Other advantages and benefits of the object selection system will become apparent from the following description. In addition, as shown in the previous discussion, the present disclosure uses various terms to describe the features and advantages of the object selection system. Before describing the object selection system with reference to the following figures, additional details about the meaning of these terms are now provided.

[0050] As used herein, the term "digital image" (or simply "image") refers to a digital graphics file that displays one or more objects when rendered. Specifically, an image can include one or more objects associated with any suitable object type or object class. In various embodiments, the image editing system displays the image on a computing device such as a client device. In additional embodiments, the image editing system enables a user to modify or change an image and to generate a new image. For example, the image editing system enables a user to copy an object selected in a first image to the background of a second image.

[0051] As used herein, the term "object" refers to a visual representation of a subject, concept, or sub-concept in an image. Specifically, an object refers to a set of pixels in an image that are combined to form a visual description of an item, article, part of an item, component, or element. Objects can correspond to a variety of categories and concepts. For example, objects include professional objects, conceptual objects, objects from known categories and unknown object categories (e.g., objects that are not used to train any of the object detection neural networks). In some embodiments, an object includes multiple instances of the object. For example, an image of a rose bush includes multiple instances of roses. In one or more embodiments, an object includes a sub-object, part, or component. For example, a person's face or leg can be an object that is part of another object (e.g., the person's body). As another example, a shirt is an object that can be part of another object (e.g., a person).

[0052] As described above, the object selection system can accurately detect and automatically select objects within an image based on a query string. As used herein, the term "query string" refers to a text string of one or more words that indicates a target object. The query string may include a noun representing the query object. Typically, the object selection system receives a query string when a user requests the object selection system to automatically select an object in an image. In some embodiments, the query string is submitted as a text string. In alternative embodiments, the object selection system detects alternative user input such as voice data and converts the alternative user input into text to obtain the query string. In various embodiments, the query string is a selection query.

[0053] As used herein, the term "object mask" or "segmentation mask" or "object segmentation" refers to an indication of a plurality of pixels that delineate an object. For example, an object mask may include a segmentation boundary (e.g., a boundary line or curve indicating the edge of one or more objects) or a segmentation mask (e.g., a binary mask that identifies pixels corresponding to an object). Generating an object mask is sometimes referred to as "selecting" a target object (i.e., identifying pixels that represent a target object).

[0054] As used herein, the term "approximate boundary" refers to an indication of an area that includes an object that is larger and / or less precise than an object mask. In one or more embodiments, the approximate boundary may include at least a portion of the query object and a portion of the image that does not include the query object. The approximate boundary may include any shape, such as a square, rectangle, circle, ellipse, or other outline around the object. In one or more embodiments, the approximate boundary includes a bounding box.

[0055] As just mentioned, the query string may include a query object. The term "query object" refers to the object in the query string that the user requests to be detected and selected. For example, a noun in the query string represents the query object. In additional embodiments, the query string includes additional words, such as adjectives and adverbs that indicate the properties of the query object. As used herein, the term "object properties" refers to descriptive terms that further identify the query object. Examples of object properties include color, size, length, shape, positioning, location, pattern, composition, expression, emotion, rigidity and / or flexibility.

[0056] The term "object selection pipeline" refers to a collection of components and actions for detecting and selecting a query object in an image. In various embodiments, the object selection system utilizes a subset of the components and actions in the object selection pipeline to detect and select a query object in an image, where the output from one component is provided as an input to another component. The components and actions may include neural networks, machine learning models, heuristic models, and / or functions. In addition, the components and actions in the object selection pipeline may be interchangeable, removable, replaceable, or upgradeable, as described in further detail below.

[0057] As used herein, the term "object attribute detection model" refers to one or more methods, algorithms, actions, and / or steps for detecting instances of objects. In some embodiments, the object attribute detection model may utilize machine learning techniques, such as the following in combination with Fig.14 In one or more embodiments, the object attribute detection model does not utilize machine learning techniques. For example, the object attribute detection model utilizes non-machine learning heuristics. Figures 10 to 13 Provides examples of object property detection models.

[0058] As described above, in various embodiments, the object selection system can employ machine learning and various neural networks. As used herein, the term "machine learning" refers to the process of constructing and implementing algorithms that can learn from data and make predictions about the data. Typically, machine learning can operate by building a model from example inputs of image exposure training pairs in a training data set such as images to make data-driven predictions or decisions. Machine learning can include neural networks (e.g., natural language processing neural networks, object detection neural networks, object proposal neural networks, unknown object category detection neural networks, region proposal neural networks, concept embedding neural networks, object mask neural networks, object classification neural networks, and / or object attribute detection neural networks), data-based models (e.g., natural language processing models, object recognition models, filtering models, and / or object attribute detection models), or a combination of networks and models.

[0059] As used herein, the term "neural network" refers to a machine learning model that can be tuned (e.g., trained) based on inputs to approximate unknown functions. Specifically, the term neural network can include a model of interconnected neurons that communicate and learn to approximate complex functions and generate outputs based on multiple inputs provided to the model. For example, the term neural network includes an algorithm (or set of algorithms) that implements a deep learning technique that uses a set of algorithms to tune the parameters of a neural network using supervised data to model high-level abstractions in the data. Examples of neural networks include convolutional neural networks (CNNs), regional CNNs (R-CNNs), faster R-CNNs, Mask R-CNNs, and single shot detection (SSD).

[0060] Now referring to the accompanying drawings, Figure 1 1 shows a schematic diagram of an environment 100 in which an object selection system 106 can operate according to one or more embodiments. Figure 1 As shown, environment 100 includes client device 102 and server device 110 connected via network 108. Fig.18 Provides additional details about computing devices (e.g., client device 102 and server device 110). Additionally, Fig.18 Additional details regarding networks are also provided, such as network 108 as shown.

[0061] although Figure 1 A specific number, type, and arrangement of components within environment 100 are shown, but various additional environment configurations are possible. For example, environment 100 may include any number of client devices. As another example, server device 110 may represent a collection of connected server devices. As another example, client device 102 may communicate directly with server device 110, bypassing network 108 or utilizing a separate and / or additional network.

[0062] As shown, environment 100 includes client device 102. In various embodiments, client device 102 is associated with a user (e.g., a user client device), such as a user who requests automatic selection of an object in an image. Client device 102 may include image editing system 104 and object selection system 106. In various embodiments, image editing system 104 implements object selection system 106. In alternative embodiments, object selection system 106 is separate from image editing system 104. Although image editing system 104 and object selection system 106 are shown on client device 102, in some embodiments, image editing system 104 and object selection system 106 are remote from client device 102 (e.g., on server device 110), as further explained below.

[0063] In general, the image editing system 104 facilitates the creation, modification, sharing, and / or deletion of digital images within an application. For example, the image editing system 104 provides various tools related to image creation and editing (e.g., photo editing). For example, the image editing system 104 provides selection tools, color correction tools, and image manipulation tools. In addition, the image editing system 104 can operate in conjunction with one or more applications to generate or modify images. For example, in one or more embodiments, the image editing system 104 can operate in conjunction with, for example, digital design applications or other image editing applications.

[0064] In some embodiments, the image editing system 104 provides an intelligent image editing assistant that performs one or more automatic image editing operations for the user. For example, given an image of three people, the user requests the image editing system to "remove the tallest man." As part of fulfilling the request, the image editing system 104 utilizes the object selection system 106 to automatically select the tallest man. The image editing system 104 can then utilize additional system components (e.g., a hole filling neural network) to remove and replace the selected tallest person (e.g., the detected query object).

[0065] As described above, the image editing system 104 includes an object selection system 106. As described in detail below, the object selection system 106 accurately detects and automatically selects objects in an image based on a user's request (e.g., a query string provided by the user). In many embodiments, the object selection system 106 utilizes an object selection pipeline to determine which object detection neural networks to utilize based on a query object, and to determine which additional neural networks and / or models to use to select a particular requested query object.

[0066] As shown, environment 100 also includes server device 110. Server device 110 includes object selection server system 112. For example, in one or more embodiments, object selection server system 112 represents and / or provides similar functionality as described herein in conjunction with object selection system. In some embodiments, object selection server system 112 supports object selection system 106 on client device 102.

[0067] Indeed, in one or more embodiments, server device 110 may include all or a portion of object selection system 106. Specifically, object selection system 106 on client device 102 may download an application from server device 110 (e.g., an image editing application from object selection server system 112) or a portion of a software application.

[0068] In some embodiments, the object selection server system 112 may include a web hosting application that allows the client device 102 to interact with content and services hosted on the server device 110. In one or more embodiments, for illustration, the client device 102 accesses a web page supported by the server device 110, which automatically selects objects in an image based on a query string provided by a user via the client device 102. As another example, the client device 102 provides an image editing application that provides the image and the query string to the object selection server system 112 on the server device 110, which then detects query object instances and provides object masks of the detected query object instances back to the client device 102. Then, using the object masks, the image editing application on the client device 102 selects the detected query object instances.

[0069] As mentioned above, Figure 1 An example environment 100 is shown in which the object selection system 106 may operate. Turning to the next figure, Figure 2 The present invention provides an overview of the process of automatically selecting object instances in an image using an object selection system. Specifically, Figure 2 A series of acts 200 are shown, illustrating a schematic diagram of automatically detecting and selecting a target query object instance in an image in accordance with one or more embodiments. In various embodiments, the object selection system 106 performs the series of acts 200 .

[0070] like Figure 2 As shown, a series of actions 200 includes an action 202 of the object selection system 106, which identifies a query string indicating a target object in an image. For example, a user edits an image using an image editing program. While editing an image, the user wishes to select a specific object within the image. Therefore, the object selection system 106 provides the user with a graphical interface that enables the user to enter a query string (i.e., a selection query) requesting automatic selection of an object. Figure 2 As shown, in conjunction with action 202 , the user provides a query string of “middle car” to be selected from images of three cars.

[0071] In one or more embodiments, as part of act 202, object selection system 106 may utilize natural language processing to identify a query object from the query string. For example, object selection system 106 determines that the query string includes a query object of "car". In additional embodiments, object selection system 106 may determine one or more object properties from the query string corresponding to the query object. For example, object selection system 106 determines that the query string includes an object property of "middle".

[0072] In response to receiving the query string, the object selection system 106 performs action 204, which utilizes an object detection neural network to determine multiple instances of the query object included in the selection query. For example, the object selection system 106 utilizes one or more object detection neural networks to detect each instance or occurrence of the query object within the image. In practice, when multiple instances of the query object are included in the image, the object selection system 106 can detect each of the multiple instances. As shown, action 206 shows the object detection neural network detecting each of the three cars in the image.

[0073] As shown, the series of actions 200 includes an action 206 of the object selection system 106, which determines an object attribute detection model based on the object attributes included in the query string. Specifically, when the object attribute is detected, the object selection system 106 can analyze the object attribute to determine an object attribute detection model corresponding to the object attribute. As described above, based on the analysis of the object attribute, the object selection system 106 can determine to utilize a dedicated object attribute detection model, the dedicated object attribute detection model corresponding to color, size, length, shape, positioning, location, pattern, composition, expression, emotion, rigidity, and / or flexibility. In some embodiments, the object selection system 106 determines to select a general (i.e., non-dedicated) object attribute detection neural network, as further described below.

[0074] In act 208, the object selection system 106 may use the determined object property detection model and the object properties to determine that the target query object instance is the target query object. To illustrate, as shown in conjunction with act 208, the object selection system 106 selects the middle car based on the object property of "middle." In various embodiments, the object selection system 106 detects the middle car from other detected cars in the image using the object localization detection model.

[0075] When selecting a target instance of a query object that satisfies the object properties in the query string, the object selection system 106 can provide the target query object instance to the user. For example, the object selection system 106 can automatically select the target query object instance within the image. In other words, the object selection system 106 can generate and provide an object mask of the middle car as shown. In some embodiments, the object selection system 106 automatically selects the target query object instance within the image editing application in response to the selection query.

[0076] In some embodiments, object selection system 106 may detect multiple target instances of a query object. Specifically, if more than one of the multiple query object instances of the query object satisfies the object properties in the query string, object selection system 106 may automatically select those query object instances as target query object instances. To illustrate, if the image depicts a parking lot of cars and the query string is "red cars", object selection system 106 may automatically detect and select all red cars.

[0077] In various embodiments, the query string includes more than one object attribute. In these embodiments, the object selection system 106 may determine a plurality of object attribute detection models corresponding to the plurality of object attributes (e.g., repeating action 206). Further, the object selection system 106 may utilize each of the plurality of object attribute detection models to identify a target query object instance that satisfies each of the plurality of object attributes (e.g., repeating action 208). To illustrate, for the query string "top, red ball", the object selection system 106 may utilize a color detection model to find one or more red balls, and utilize a position detection model to find the highest located ball(s). The object selection system 106 then identifies one or more instances of the detected red and highest located ball.

[0078] FIG. 3A to FIG. 3B A client device 300 is shown having a graphical user interface 302 showing selected query object instances in an image based on a selection request in accordance with one or more embodiments. FIG. 3A to FIG. 3B The client device 300 in the embodiment may represent the above Figure 1 The client device 102 is described. For example, the client device 300 includes an image editing application that implements the image editing system 104 with the object selection system 106. For example FIG. 3A to FIG. 3B The graphical user interface 302 in may be generated by an image editing application.

[0079] like Figure 3A As shown, graphical user interface 302 includes image 304 within an image editing application. Image 304 may correspond to the above combined Figure 2 For ease of illustration, image 304 is simplified to show three cars without background or other objects.

[0080] In response to the user selecting the option to have the object automatically selected, the object selection system 106 may provide an object selection request interface 306. For example, the object selection system 106 provides the object selection request interface 306 as a selection tool within an image editing application. As shown, the object selection request interface 306 may include a text field in which the user may enter a query string (i.e., "middle car"). The object selection request interface 306 also includes an optional option for confirming or canceling the selection request. In some embodiments, the object selection request interface 306 includes additional elements, such as an optional option for capturing audio input from the user indicating the query string.

[0081] Based on receiving a selection request including a query string (i.e., "middle car"), object selection system 106 can accurately detect and automatically select the query object. Specifically, object selection system 106 can detect each instance of the query object in image 304 (e.g., each of the three cars), and identify the specific or target instance specified in the query string (e.g., the "middle" car). To illustrate, Figure 3B The result of object selection system 106 detecting and selecting middle car 308 within image 304 in response to the selection request is shown. Once selected, the image editing application may enable the user to edit, copy, cut, move, and / or otherwise modify the selected object.

[0082] As shown, the object selection system 106 enables the user to provide a query string within the object selection request interface 306. In some embodiments, the object selection system 106 enables the user to provide audio or other input to indicate the query string. For example, the client device 300 may include a microphone that captures the user speaking the query string and speech-to-text processing to convert the query string into text. In addition, as described below, in some embodiments, the object selection system 106 enables the user to provide additional input such as a mouse or touch gesture to further assist the object selection system 106 in automatically selecting a target query object.

[0083] Now turn to Figures 4 to 15 , provides additional details about the object selection system 106 utilizing the object selection pipeline to automatically select and accurately detect the requested object in the selection request. Specifically, Figure 4 An example embodiment of an object selection pipeline is shown. Figures 5 to 15 An extension of an object selection pipeline for selecting a target instance of a detected query object together with a corresponding graphical user interface is described.

[0084] As mentioned above, Figure 4A schematic diagram of an object selection pipeline 400 is shown in accordance with one or more embodiments. In various embodiments, the object selection system 106 performs the actions included in the object selection pipeline 400. In alternative embodiments, an image editing system and / or an image editing application may perform one or more of the included actions.

[0085] As shown, the object selection pipeline 400 includes an action 402 of obtaining an image (i.e., a digital image) by the object selection system 106. For example, as previously described, the object selection system 106 can detect a user selecting an image to be loaded into an image editing application. Typically, an image includes one or more objects. For example, an image can include objects of various types and categories.

[0086] Additionally, the object selection pipeline includes an action 404 of obtaining a query string by the object selection system 106. For example, the object selection system 106 provides an object selection request interface (e.g. Figure 3A As shown), a user may enter one or more words indicating an object that they wish the object selection system to automatically select. As described above, in some embodiments, the object selection system 106 may support additional or alternative forms of user input, such as audio input or touch input.

[0087] Next, the object selection pipeline 400 includes an action 406 of the object selection system 106 to analyze the query string to identify the query object and the corresponding object attributes. As described above, when multiple instances of the query object are included in the image, the query string typically includes the object attributes of the query object. In this way, the object attributes indicate which target instance of the multiple query object instances requested by the user is automatically selected. Additionally, the object selection system 106 can parse the query string and utilize natural language processing to identify the query object. With respect to the following Figure 5 Additional details regarding act 406 are provided.

[0088] As shown, the object selection pipeline 400 includes an action 408 of the object selection system 106, which utilizes an object detection network to detect multiple instances of a query object. As used herein, an object detection network may correspond to one or more types of object detection neural networks that the object selection system 106 utilizes to select a query object and / or multiple query object instances in an image. Figures 6 to 7C Additional details regarding act 408 are provided.

[0089] As part of action 408, object selection system 106 may create an approximate boundary around each query object instance of the detected query object. In some embodiments, object selection system 106 may generate a bounding box around each of the plurality of query object instances. In some cases, the bounding box may provide a set of pixels or a subset of pixels within the image that include the corresponding query object instance.

[0090] like Figure 4 As shown, the object selection pipeline 400 includes an action 410, which uses an object attribute detection model to identify a target instance of a detected query object from multiple query object instances. For example, the object selection system 106 can determine and select an object attribute detection model corresponding to the object attribute indicated in the query string. Then, the object selection system 106 can use the object attribute detection model to detect from the query string that the target query object instance includes the object attribute, or that other query object instances lack the object attribute. Specifically, the object selection system 106 can then use the object attribute detection model to analyze the bounding boxes corresponding to the multiple detected instances of the query object to determine whether they include the object attribute. In one or more embodiments, the object selection system 106 can generate an object mask for each instance of the multiple query objects, and then use the object mask corresponding to each query object instance to determine whether a given query object instance is a target query object (i.e., has the object attribute).

[0091] As also shown, the object selection pipeline 400 includes an action 412 of the object selection system 106 to generate one or more object masks for the query object instance using the object mask neural network. For example, in various embodiments, the object selection system 106 may provide a bounding box corresponding to the identified target query object instance to the object mask neural network, which generates a mask for the detected target query object. If multiple bounding boxes are provided, the object selection system 106 may generate multiple object masks from the multiple bounding boxes using the object mask neural network (e.g., one object mask for each instance of the detected target query object).

[0092] When generating an object mask for a target query object, the object mask neural network can segment the pixels in the detected query object from other pixels in the image. For example, the object mask neural network can create a separate image layer that sets the pixels corresponding to the query object instance to positive (e.g., binary 1) while setting the rest of the pixels in the image to neutral or negative (e.g., binary 0). When this object mask layer is combined with the image, only the pixels of the query object instance are visible. In effect, the generated object mask can provide a segmentation that enables selection of the query object instance within the image.

[0093] The object mask neural network may correspond to one or more deep neural networks or models that select objects based on bounding box parameters corresponding to objects within an image. For example, in one or more embodiments, the object mask neural network utilizes techniques and methods found in Ning Xu et al., "Deep GrabCut for Object Selection," published on July 14, 2017, the entire contents of which are incorporated herein by reference. For example, the object mask neural network may utilize a deep grad cut approach instead of a saliency mask transfer. As another example, the object mask neural network can utilize techniques and methods found in U.S. Patent Application Publication No. 2019 / 0130229, titled “Deep Salient Content Neural Networks for Efficient Digital Object Segmentation,” filed on October 31, 2017; U.S. Patent Application No. 16 / 035,410, titled “Automatic Trimap Generation and Image Segmentation,” filed on July 13, 2018; and U.S. Patent No. 10,192,129, titled “Utilizing Interactive Deep Learning To Select Objects In Digital Visual Media,” filed on November 18, 2015, the entire contents of each of which are incorporated herein by reference.

[0094] Furthermore, for embodiments in which the object mask neural network performs instance-level segmentation (e.g., semantic segmentation), the object mask neural network can utilize techniques and methods found in U.S. Patent Application Publication No. 2018 / 0108137, entitled “Instance-Level Semantic Segmentation System,” filed on October 18, 2016. Additionally or alternatively, the object mask neural network can utilize techniques and methods found in U.S. Patent No. 9,129,191, entitled “Semantic Object Selection,” issued on September 8, 2015, the entire contents of each of which are incorporated herein by reference.

[0095] like Figure 4As shown in the object selection pipeline 400 of , the object selection system 106 may proceed from act 408 to act 410 or act 412. Specifically, depending on which object attribute detection model the object selection system 106 selects, the object selection system 106 may determine when to generate one or more object masks. To elaborate, in multiple embodiments, the object selection system 106 utilizes an object attribute detection model that may detect a target query object instance using approximate boundaries (e.g., bounding boxes) of multiple query object instances. In these embodiments, the object selection system 106 may proceed from act 408 to act 410 to first detect the target query object instance. Then, the object selection system 106 may proceed to act 412 to generate an object mask only for the target query object instance without generating additional object masks for other query object instances of the multiple query object instances. In these embodiments, the object selection system 106 prevents wasting processing resources in generating object masks for query object instances of the detected query object that the user does not want to select.

[0096] In alternative embodiments, object selection system 106 may determine to utilize an object attribute detection model that uses a separate object mask for each of a plurality of query object instances to detect a target query object instance. For example, an approximate boundary surrounding a query object instance includes additional pixels that do not belong to the query object instance, and these additional pixels may negatively impact the accuracy of the query object instance detection process. Therefore, in these embodiments, object selection system 106 may proceed from action 408 to action 412 to first generate an object mask for each of a plurality of query object instances. Object selection system 106 may then proceed to action 410 to detect a target query object instance using the selected object attribute detection model based only on the pixels belonging to each query object instance. Figures 8 to 15 Provides additional details on determining an object property detection model and identifying target query object instances.

[0097] In addition, the object selection pipeline 400 includes an action 414 of the object selection system 106 providing the selected query object instance within the image to the user. For example, the object selection system 106 may provide the selected target instance (or multiple query object instances) of the query object to a client device associated with the user. In some cases, the object selection system 106 may automatically select the target query object instance within the image for the user within the above-mentioned image editing application.

[0098] In various embodiments, many components of the object selection pipeline 400 are interchangeable with updated versions and new components. Thus, when a failure occurs, the object selection system 106 can identify and update the source of the failure. Additionally, the object selection system 106 can also add additional components to the object selection pipeline to improve the performance of detecting objects in an image.

[0099] More specifically, upon detecting the location of a fault within the object selection pipeline 400, the object selection system 106 can upgrade or replace the faulty component. For example, the object selection system 106 can replace the faulty component with an improved version. Similarly, when an updated version is available, the object selection system 106 can replace the old component with a new component of the updated version. In addition, the object selection system 106 can upgrade and / or replace components without changing, retraining, reconfiguring, and / or modifying other components of the object selection pipeline. Additionally, the object selection system 106 can continue to utilize the object selection pipeline until another fault is detected.

[0100] Now, moving on to the next figure, Figure 5 A flow chart of analyzing a query string to identify a query object according to one or more embodiments is shown. As described above, Figure 5 Corresponding to the above combination Figure 4 The described object selects action 406 of the pipeline 400. Specifically, Figure 5 Various actions 502 - 508 are included that provide additional details regarding action 406 of object selection pipeline 400 .

[0101] As shown, action 406 may include action 502 of the object selection system 106 parsing the query string to identify one or more input terms. For example, in various embodiments, the object selection system 106 separates the query string into individual terms. For example, the object selection system 106 assigns each term in the query string to an element of a vector corresponding to the query string. Moreover, as described above, in some embodiments, the query string consists of a single term.

[0102] As also shown, action 406 may include action 504 of the object selection system 106, which utilizes natural language processing to determine a term category for each of the one or more input terms. In practice, upon parsing each term in the query string, the object selection system 106 may classify and assign each input term in the query string to a term category. In various embodiments, the term category includes a part of speech (e.g., a noun, a proper noun, a verb, an article, an adjective, an adverb, a pronoun, a preposition, or a conjunction).

[0103] In one or more embodiments, the object selection system 106 utilizes a machine learning model trained based on natural language processing to identify the term category of each term in the query string. For example, the object selection system 106 trains and / or utilizes a long short-term memory (LSTM) neural network to identify a term category for each term in the query string. In alternative embodiments, the object selection system 106 utilizes other methods to determine the term category of the input term in the query string.

[0104] As shown, action 406 may include action 506 of the object selection system 106, which identifies nouns that indicate the query object. More specifically, the object selection system 106 may identify nouns in the query string when assigning a term category to each of the terms. First, the nouns in the query string correspond to the query object. Typically, when the query string contains a single term, the term is a noun that corresponds to the query object.

[0105] In some cases, the query string includes multiple nouns. In these cases, the object selection system 106 can determine whether the two nouns are connected to indicate a single query object (e.g., "German" and "Shepard"). In an alternative embodiment, the object selection system 106 can determine whether the query string includes multiple query objects for the object selection system 106 to automatically select.

[0106] Additionally, action 406 may include action 508 of the object selection system 106, which identifies adjectives that indicate object properties of the query object. Similar to identifying nouns in a query string, the object selection system 106 may also identify adjectives in the query string as object properties. Typically, a query string with multiple terms will include a query object and one or more adjectives that specify specific properties of the query object. As described below, an object property may specify the color, size, location, shape, material, position, rigidity, or expression of the query object.

[0107] In one or more embodiments, the object selection system 106 may also identify adverbs or other parts of speech (i.e., word categories) in the query string. In some embodiments, the object selection system 106 may assign the adverb an as an object attribute of the query object. Additionally, in one or more embodiments, the object selection system 106 may group adverbs with corresponding adjectives (e.g., "very high"). In alternative embodiments, the object selection system 106 may ignore words that are not nouns or adjectives.

[0108] Similarly, in one or more embodiments, the object selection system 106 can filter out terms of the query string, such as articles (e.g., "the," "an," "the"). In some embodiments, the object selection system 106 can identify connections that indicate the possibility that multiple query objects are requested and / or multiple object properties are provided. Similarly, in some embodiments, the object selection system 106 identifies negative terms, such as "not," "except," or "without," which can indicate exclusion of specific objects when selecting query objects.

[0109] Go to Figure 6 , Figure 6 is a flow chart of detecting multiple query object instances in an image using an object detection network according to one or more embodiments. As described above, Figure 6 The object selection system 106 detects multiple instances of the query object using the object detection network in action 408. As also described above, the object selection system 106 arrives at action 408 based on identifying the query object within a query string of a selection query, where the selection query corresponds to an image. Specifically, Figure 6 Various actions 602 - 610 are included that provide additional details about action 408 of object selection pipeline 400 .

[0110] As shown, action 408 includes action 602 of object selection system 106 identifying an object detection neural network based on the query object. In some embodiments, object selection system 106 utilizes a general object detection neural network to detect multiple instances of the query object in the image. In alternative embodiments, object selection system 106 determines to utilize a more specific, dedicated object detection neural network. In fact, an object detection neural network can be specially trained to detect a specific object type or object category.

[0111] For illustration, examples of object detection neural networks include specialized object detection neural networks (e.g., sky detection neural networks, face detection neural networks, body detection neural networks, skin detection neural networks, waterfall detection neural networks), object-based concept detection neural networks, known object category detection neural networks, unknown object category detection neural networks. Examples of object detection neural networks may also include subnetworks and / or supporting object detection networks, such as object proposal neural networks, region proposal neural networks, and concept embedding neural networks. Additional description of object detection neural networks is provided in U.S. patent application Ser. No. 16 / 518,810 (Attorney Docket No. 20030.257.2), filed on July 22, 2019, entitled “AUTOMATICALLY DETECTING USER-REQUESTED OBJECTS IN IMAGES,” the entire contents of which are incorporated herein by reference.

[0112] like Figure 6 As shown, action 408 may include action 604 of object selection system 106 detecting candidate objects in the image using an object detection neural network. In one or more embodiments, object selection system 106 detects a plurality of candidate objects in the image using the object detection neural network, wherein at least some of the candidate objects match the query object. In alternative embodiments, each of the candidate objects may match the query object.

[0113] Additionally, action 408 may include action 606 of the object selection system 106 generating a bounding box for each candidate object. For example, as part of detecting the candidate objects, the object detection neural network creates an approximate bounding box around each detected candidate object. In some embodiments, the object detection neural network may process sub-portions or patches of the image to identify smaller candidate objects within the image that may not be detected using the whole image object detection method. Thus, the object detection neural network may generate a plurality of bounding boxes, each of which indicates a candidate object in the image that may potentially be a query object.

[0114] As shown, action 408 may include action 608 where the object selection system 106 detects a plurality of query object instances from the candidate objects. In some embodiments, the object detection neural network generates a label for each of the bounding boxes and associates the label with the query object. In other embodiments, the object detection neural network further processes each candidate object to determine whether it corresponds to the query object. As described in U.S. Patent Application Serial No. 16 / 518,810 referenced above, the object detection neural network may utilize various techniques to identify a plurality of query object instances from the candidate objects.

[0115] FIG. 7A to FIG. 7C 702 is shown, according to one or more embodiments, illustrating that the object selection system 106 uses a known object category detection neural network to detect a query object. For ease of explanation, FIG. 7A to FIG. 7C The client device 300 described above is included. For example, the client device 300 includes an image editing application that implements the image editing system 104 using the object selection system 106 .

[0116] like Fig. 7A As shown, graphical user interface 702 includes image 704 within an image editing application. Image 704 shows a woman and a turtle in front of a row of trees. As also shown, the image editing application includes various tools (e.g., a vertical toolbar) with selection options and other image editing options. Additionally, graphical user interface 702 includes object selection interface 706, as described above in conjunction with Figure 3ADescribed, where the user provides a "tree" query object.

[0117] As previously described, when the user provides a query string, the object selection system 106 may first identify the query object within the query string as "tree". In addition, the object detection neural network may determine and utilize the object detection neural network to automatically detect one or more instances of the query object in response to the selection query. For example, if "tree" is a known object, the object selection system 106 may utilize the known object class detection neural network to detect any instances of trees (i.e., the query object) within the image.

[0118] To illustrate, Figure 7B It is shown that the object selection system 106 can utilize an object detection neural network to identify candidate objects 708a-708f within the image 704. As part of detecting the candidate objects 708a-708f, the object selection system 106 generates bounding boxes around the objects and marks each of the bounding boxes with a label. As shown, the object selection system 106 detects candidate objects of trees 708a-708d, a woman 708e, and a turtle 708f.

[0119] As described above, object selection system 106 can determine which of candidate objects 708a-708f correspond to the query object of "tree." For example, object selection system 106 determines that trees 708a-708d correspond to the query object, while woman 708e and turtle 708f do not correspond to the query object. In addition, object selection system 106 can determine that image 704 includes multiple query object instances of a detected tree.

[0120] To illustrate, Figure 7C The object selection system 106 is shown detecting multiple query object instances 710a-710d of the tree. In additional embodiments, as described in detail below, if the query string includes object properties corresponding to the query object, the object selection system 106 can further detect a target query object instance from the multiple query object instances 710a-710d.

[0121] Although not shown, in some embodiments, in response to a selection query, the object selection system 106 may select multiple query object instances 710a-710d and provide them to the user. In practice, the object selection system 106 may provide bounding boxes of the multiple query object instances 710a-710d to the object mask neural network described above. In addition, as described above, the object selection system 106 may utilize individually generated object masks to individually select trees for the user within the image editing application.

[0122] Notice, Figure 7B and 7CThe object selection system 106 is shown using an object detection neural network to detect multiple query object instances 710a-710d. In many embodiments, the object selection system 106 does not display these illustrated actions to the user. Instead, the object selection system 106 appears to automatically detect and accurately select multiple query object instances 710a-710d in response to the user's selection query. In alternative embodiments, the object selection system 106 displays one or more of these actions to the user. For example, the object selection system 106 displays bounding boxes of the multiple query object instances 710a-710d, such as Figure 7C shown.

[0123] Steering Figure 8 , shows a flow chart of detecting a target instance of a query object according to one or more embodiments. As described above, Figure 8 Corresponding to action 410, as above combined Figure 4 As described above, the action 410 uses the object attribute detection model included in the object selection pipeline 400 to identify the target instance of the detected query object from multiple query object instances. Specifically, Figure 8 Various actions 802 - 810 are included that provide additional details regarding action 410 of object selection pipeline 400 .

[0124] As described above, in various embodiments, the object selection system 106 may detect more than one instance of a query object in an image. As provided in the above example, if the object selection system 106 detects the query object "tree" in a landscape image, the object selection system 106 may detect multiple trees. In many embodiments, the object selection system 106 may generate and provide an individual selection for each instance of the query object detected. In an alternative embodiment, the object selection system 106 may generate a single aggregate selection of multiple instances of the query object detected within the image editing application.

[0125] In one or more embodiments, the object selection system 106 may identify and select one or more instances of the detected query object. Specifically, the object selection system 106 may select one or more target query object instances based on object attributes indicated in the query string. For illustration, Figure 8 Acts 802 - 810 of act 410 in further describe an embodiment of the object selection system 106 that selects a target instance of a detected query object from among a plurality of query object instances of the detected query object.

[0126] As shown, act 410 may include act 802 in which object selection system 106 identifies multiple instances of a query object. As described above, object selection system 106 may detect more than one instance of a query object in an image. For example, object selection system 106 detects multiple instances of a dog, a mountain, a road, a car, a sign, or a person within various images.

[0127] In one or more embodiments, the object selection system 106 may identify adjectives that indicate properties of the query object (ie, object properties). Figure 5 As explained, in one or more embodiments, the object selection system 106 can identify one or more adjectives in the query string as object attributes. Typically, a query string with multiple terms will include a query object along with one or more adjectives that specify specific attributes of the query object.

[0128] As shown, action 410 may include action 804 in which the object selection system 106 determines a property type based on the object properties. For example, as described above, the object properties may specify the color, size, length, positioning, shape, pattern, material, location, depth, rigidity, protrusion, body posture, or facial expression of the query object. Thus, the object selection system 106 may analyze each object property to determine whether it corresponds to a known or unknown object property type.

[0129] In one or more embodiments, the object selection system 106 can perform a table lookup to determine whether an object property corresponds to a particular object property type. For example, the lookup table can include a list of colors by name and indicate that these object properties are associated with colors. Additionally, the object selection system 106 can include similar entries for other object property types, such as shape, material, and positioning. In an alternative embodiment, the object selection system 106 trains an object property type neural network to predict the object property type of an object property.

[0130] As shown, act 410 may include act 806 of object selection system 106 selecting an object property detection model based on the object property type. In various embodiments, object selection system 106 maintains multiple object property models corresponding to various object property types. Thus, upon detecting a particular object property type, object selection system 106 may select a corresponding object property model that will accurately process the object properties in the query string to select the correct instance of the detected query object.

[0131] To illustrate, if the object property type is color, the object selection system 106 may select an object color detection model to identify and select the color (i.e., object property) provided in the query string. Similarly, if the object property type is location, the object selection system 106 may select an object location detection model to identify and select detected instances of the query object having the location indicated in the query string. Additionally, the object selection system 106 may select additional specialized and / or general object property detection models based on the object property type.

[0132] As shown, action 410 may include optional action 808 in which object selection system 106 generates separate object masks for multiple query object instances. As described above, in various embodiments, object selection system 106 may identify and select target instances based on object masks. For object attribute types corresponding to these embodiments (e.g., color, material, relative positioning, or shape), object selection system 106 may increase the accuracy of selection by considering only pixels belonging to the object itself rather than considering pixels of the background surrounding the instance in the approximate boundary of a bounding box. Therefore, in these embodiments, object selection system 106 may perform optional action 808 before proceeding to action 810 (described below).

[0133] In alternative embodiments, object selection system 106 may determine a target query object instance based on the approximate boundaries corresponding to each of the plurality of query object instances. In these embodiments, object selection system 106 may perform action 810 (described below) before generating an object mask for the detected target query object instance. In practice, for many object attribute types (e.g., location, position, expression, depth, or size), object selection system 106 may select the target query object instance using the corresponding approximate boundaries (e.g., a bounding box) before generating individual instance object masks. By not generating object masks that will not be selected as the target query object instance (e.g., not generating additional object masks for non-target query object instances of the plurality of query object instances), object selection system 106 may reduce processing and selection time.

[0134] As described above, act 410 may include act 810 of the object selection system 106 identifying a target instance of the query object corresponding to the object attribute using the selected object attribute detection model. For example, if the object attribute is "blue" (e.g., the query string is "blue balloon"), the object selection system 106 identifies each detected balloon in the image that is blue using the object color detection model. With respect to the following Figures 10 to 14 , provides additional details on various object attribute detection neural networks and models.

[0135] 9A to 9DA graphical user interface for detecting target instances of a query object using an object attribute detection neural network according to one or more embodiments is shown. For ease of illustration, 9A to 9D The client device 300 described above is included. For example, the client device 300 includes an image editing application that implements the image editing system 104 using the object selection system 106 .

[0136] like Fig. 9A As shown, graphical user interface 902 includes image 904 within an image editing application. Image 904 shows three men talking. As also shown, the image editing application includes various tools (e.g., a vertical toolbar) with selection options and other image editing options. Additionally, graphical user interface 902 includes object selection request interface 906, as described above in conjunction with Figure 3A As described, where the user provides the query string "striped man".

[0137] exist Fig. 9B , the object selection system 106 can detect three instances of the detected query object (i.e., "person") in the image 904 indicated by the three bounding boxes 908a-908c. For example, when a user receives a query string from the object selection request interface 906, the object selection system 106 can utilize the object selection pipeline 400 to utilize one or more object detection neural networks to detect multiple instances of the detected query object, as described above.

[0138] In addition, the object selection system 106 can perform instance segmentation on each of the detected instances of the query object, such as Fig. 9C For example, the object selection system 106 utilizes an object mask neural network (e.g. Figure 4 4) to generate a separate object mask for each of the detected persons in the image. Fig. 9C As shown, the graphical user interface 902 includes a first object mask 910a, a second object mask 910b, and a third object mask 910c.

[0139] Additionally, the object selection system 106 can detect that the query string includes an object attribute (i.e., "stripes"). For example, the object selection system 106 can parse the query string to determine an adjective that corresponds to the object attribute of the query object. Additionally, the object selection system 106 can determine an object attribute type (e.g., clothing pattern) for the object attribute.

[0140] In various embodiments, based on the object attributes and / or object attribute types, the object selection system 106 may select an object attribute detection neural network or model to detect instances that best correspond to the object attributes requested by the user. For example, the object selection system 106 may detect an object pattern detection neural network to identify each of the instances of wearing stripes. Fig.9D As shown, the object selection system can deselect the first object mask 910a and the third object mask 910c because these object masks (i.e., the set of pixels included in each object mask) do not include the object attribute of the stripe. Therefore, the object selection system 106 returns the selected target query object 912 to the user within the image editing application.

[0141] It is worth noting that Fig. 9B and Fig. 9C The object selection system 106 is shown to utilize the object attribute detection neural network to select a target instance of the detected query object. In many embodiments, the object selection system 106 does not display the corresponding action to the user. Instead, the object selection system 106 appears to automatically detect and accurately select the target query object in response to the user's query string request (e.g., the graphical user interface 902 selects the target query object from the query string). Fig. 9A Jump to Fig.9D ). In addition, as described above, in some embodiments, the object selection system 106 utilizes three bounding boxes 908a-908c ( Fig. 9B ) to select a target instance of the detected query object. Then, when selecting a target instance, the object selection system 106 generates an object mask only for the selected target query object.

[0142] Steering Fig.10 , shows a flow chart of detecting a target instance of a query object using an object color attribute detection model according to one or more embodiments. As shown and described above, Fig.10 In the above about Figure 8 The described actions 806 and 810 are expanded, which are themselves expansions of action 410 of the object selection pipeline 400. Fig.10 As shown, Fig.10 Actions 1002 - 1006 are included, which correspond to the object selection system 106 selecting and utilizing an object color detection model.

[0143] As described above, action 806 can correspond to the object selection system 106 selecting an object attribute detection model based on the object attribute type. More specifically, in various embodiments, the object selection system 106 can detect multiple instances of the detected query object indicated in the query string. In addition, the object selection system 106 determines that the query string includes one or more adjectives that specify the target query object instance. To select the target query object instance, the object selection system 106 can utilize one of the object attribute detection models that are trained or created to identify instances based on the object attribute type.

[0144] As shown, action 806 may include action 1002 in which the object selection system 106 determines that the object property type corresponds to a color. For example, the object selection system 106 analyzes the object properties (i.e., adjectives) within the query string to identify that the user is requesting to select an instance of the detected query object having a specific color. In some embodiments, the object selection system 106 determines that the object property is a color based on matching the object property with the name or description of a known color. In various embodiments, the object selection system 106 also detects that one or more object properties further specify the hue, shade, opacity, brightness, saturation, temperature, and / or color of the color (e.g., light brown, soft yellow, green, dark purple, pink, dark blue, or burnt orange).

[0145] As shown in action 1004, based on detecting that the object attribute is color, the object selection system 106 can select an object color detection model. For example, the object selection system 106 selects the object color detection model from a plurality of various object attribute detection neural networks and models. In one or more embodiments, the object color detection model includes a color classifier model described in U.S. patent application Ser. No. 16 / 518,795 (Agent Docket No. 20030.257.4), filed on July 22, 2019, entitled “CLASSIFYING COLORS OF OBJECTS IN DIGITAL IMAGES,” the entire contents of which are incorporated herein by reference.

[0146] As shown, act 810 of utilizing the selected object attribute detection model to identify a target instance of the detected query object corresponding to the object attribute may include act 1006 of object selection system 106 utilizing the object color detection model and a color (e.g., the object color attribute indicated in the query string) to identify the target instance of the detected object. In practice, object selection system 106 may utilize the object color detection model to determine the target instance from the provided color and multiple instances of the detected query object.

[0147] More specifically, in one or more embodiments, the object selection system 106 utilizes the object color detection model to convert or map the color into a mathematical representation (e.g., a color embedding vector) such as a point or region in a color space (e.g., a vector space). Additionally, the object color detection model can generate similar embeddings for pixels in a given instance of a detected query object. Furthermore, the object color detection model can compare the distance between the object color attribute and a given instance (i.e., measured using cosine similarity) to determine which instance or instances correspond to the object color attribute.

[0148] In one or more embodiments, the object color detection model generates a color embedding for all pixels of a given instance. In some embodiments, the object color detection model generates a color embedding for a subset of pixels of a given instance. Since many adjacent pixels in an image share the same color attributes, the object selection system 106 can reduce computational costs by generating a color embedding for a selected group of pixels in a given instance.

[0149] In some examples, the object selection system 106 may determine whether a pixel of a given instance is a "valid" pixel. In many embodiments, a pixel is valid if the pixel color is close (e.g., within a predetermined vector space distance) to a query color (i.e., an object color attribute). Additionally, in some embodiments, the object selection system 106 may determine whether a threshold percentage, fraction, and / or number of pixels of a given instance is valid. For example, if more than half of the pixels of a given instance are invalid (e.g., do not match the query color), the object selection system 106 filters (e.g., eliminates) the given instance from consideration as a target instance.

[0150] Similarly, if the object selection system 106 determines that the ratio of valid pixels for a given instance is satisfied (e.g., equal to or greater than 10%), the object selection system 106 may indicate the given instance as a target instance. In fact, if any of the detected instances of the query object has a valid pixel ratio above a predefined threshold, the object selection system 106 may maintain the selection of the instance.

[0151] As previously described, in some embodiments, the object selection system 106 can identify a target instance based on object properties and an approximate boundary of a detected instance of a query object. In other embodiments, the object selection system 106 utilizes an object mask of the instance. To a large extent, with respect to determining color matches, the object selection system 106 can utilize an object mask of the instance rather than a less precise approximate boundary (e.g., a bounding box) because the approximate boundary often includes background pixels having a color that does not match the object color properties.

[0152] In various embodiments, the object selection system 106 trains or otherwise obtains an object color detection model to identify instances (e.g., defined groups of pixels) corresponding to an input color. For example, the object selection system 106 generates a color space through iterative training, and trains the object color detection model to accurately map pixel colors to the color space.

[0153] To illustrate, Fig.11 A graphical user interface is shown for utilizing an object color detection model to detect target instances of a query object in accordance with one or more embodiments. Fig.11 The client device 300 described above is included. For example, the client device 300 includes an image editing application that implements the image editing system 104 using the object selection system 106 .

[0154] like Fig.11 As shown, the graphical user interface 1102 includes an image 1104 within the image editing application. The image 1104 shows a white bottle and a black bottle. Additionally, the graphical user interface 1102 includes an object selection request interface 1106, as described above in conjunction with Figure 3A Described, where the user provides a query string of "white bottle".

[0155] As previously described, the object selection system 106 utilizes the object selection pipeline 400 to detect two bottles based on the query object (i.e., "bottle") from the query string. In addition, as described above, the object selection system 106 determines that the object attribute "white" in the query string corresponds to the object color attribute. Therefore, the object selection system 106 selects and utilizes the object color detection model to identify and select the white bottle in the image (e.g., the selected target query object 1108). Therefore, as Fig.11 As shown, the object selection system 106 returns the selected target query object 1108 to the user in the image editing application.

[0156] Go to Fig.12 , shows a flow chart of using a localization object attribute model to detect a target instance of a query object according to one or more embodiments. As shown and described above, Fig.12 In the above about Figure 8 The described actions 806 and 810 are expanded, which are themselves an expansion of action 410 of the object selection pipeline 400. Also as shown, Fig.12 Actions 1202 - 1206 corresponding to the object selection system 106 selecting and utilizing an object location determination model are included.

[0157] As shown, action 806 may include action 1202 in which the object selection system 106 determines that the object attribute type corresponds to a location. For example, based on analyzing the object attributes within the query string, the object selection system 106 determines that the user is requesting selection of instances of the detected query object having a particular location within the image. In some cases, the object selection system 106 identifies the object location attribute by detecting adjectives in the query string that match a location dictionary, such as "left", "center", "right", "top", "middle", "bottom", "upper", "lower", "above", "below", "outside", "inside".

[0158] Based on detecting that the object attribute is a location, the object selection system 106 may select an object location detection model, as shown in action 1204. For example, the object selection system 106 may select the object location detection model from a plurality of object attribute detection neural networks and models, as described above.

[0159] As shown, act 810 may include act 1206 of object selection system 106 identifying a target instance of the detected query object using the object localization detection model and the localization (e.g., an object localization attribute indicated in the query string). In practice, object selection system 106 may determine the target instance from the provided localization and multiple instances of the detected query object using the object localization detection model.

[0160] To illustrate, in some embodiments, object selection system 106 may utilize an object location detection model to identify the location of each instance of a query object detected within an image. In some embodiments, the object location detection model utilizes one or more heuristics to compare the location of each query object instance. For example, the object location detection model may generate image coordinates (e.g., Cartesian coordinates) for each query object instance. Additionally, the object location detection model may add the image coordinates of each instance to a table or chart to enable object selection system 106 to compare corresponding instance locations within the image.

[0161] In various embodiments, the object location detection model determines the location of the query object instance based on its centroid. For example, the location detection model determines the location of the query object instance as its centroid coordinates. In some embodiments, the centroid corresponds to the centroid of the bounding box associated with the query object instance. In alternative embodiments, the centroid corresponds to the centroid of the object mask associated with the query object instance.

[0162] In some embodiments, the object location detection model determines the location of the query object instance based on the outer edge (e.g., the outer side of the border, the image coordinates of the corner, or the point) of the query object instance (e.g., a bounding box or an object mask). In additional embodiments, the object location detection model selects the outer edge based on an object location attribute. For example, if the object location attribute is "left", the object location detection model selects the leftmost edge or point of each query object instance. Similarly, if the object location attribute is "bottom right", the object location detection model selects the upper corner point of each query object instance where the lower edge and the right edge intersect. In addition, if the object location attribute is "top center", the object location detection model will select the middle point of the top edge of each query object instance.

[0163] In many embodiments, the object localization detection model determines the location of the query object instance relative to another object. For example, for a query string of "car to the left of the tree", the object localization detection model identifies and / or detects the tree (e.g., using an object detection neural network or other object detection model). The object localization detection model can then identify one or more query object instances to the left of the tree. In various embodiments, the object localization detection model utilizes one or more of the above methods to identify the query object instance relative to another object in the image.

[0164] After determining the location for each instance of the query object detected, object selection system 106 (and / or the object location detection model) may determine which query object instances satisfy the object location property. For example, for an object location property of "top", object selection system 106 may identify a query object instance that has a relative location (e.g., based on its image coordinates) above other query object instances in the image. As another example, for an object location property of "center", object selection system 106 may identify a query object instance that has a relative location closest to the center of the image. In some instances, for an object location property of "middle", object selection system 106 may identify a query object instance that has a relative location between at least two other query object instances.

[0165] In some embodiments, object selection system 106 determines that multiple instances of the detected query object satisfy an object location attribute. For example, if the object location attribute is "above", object selection system 106 may select each query object instance above a central horizontal line (or another vertical coordinate).

[0166] Similarly, in various embodiments, object selection system 106 may select two query object instances if the two query object instances have locations within a threshold tolerance of each other. To illustrate, if the object location attribute is "right," object selection system 106 may select the query object instance that has the rightmost location in the image (e.g., based on its x-axis image coordinate). Additionally, object selection system 106 may also select additional query object instances that have x-axis coordinates within a threshold distance (e.g., 5% of the width or 20 pixels) of the first selected query object instance.

[0167] As described above, object selection system 106 may select one or more query object instances that satisfy the object positioning property, as further described below. In practice, object selection system 106 may select one or more query object instances based on a spatial comparison of each query object instance within the image. Additionally, in some embodiments, object selection system 106 also filters out one or more query object instances that do not have a positioning that satisfies the object positioning property.

[0168] As previously explained, depending on the object location attribute, object selection system 106 can determine and identify the location of each query object instance based on their approximate boundaries before determining the object mask. For example, object selection system 106 can often perform a simple spatial comparison of query object instances with high accuracy using their corresponding bounding boxes. In this way, object selection system 106 can avoid calculating an accurate object mask for instances that do not satisfy the location attribute (i.e., the object location attribute) specified by the user in the query string.

[0169] In various embodiments, the object property corresponding to positioning may include depth. In an alternative embodiment, the object selection system 106 utilizes a separate depth property detection neural network to identify target instances corresponding to a specified depth. For example, the depth property detection neural network can generate a depth map of the image to determine the relative depth of each of the multiple instances of the query object detected. Examples of depth object properties include "front", "back", "closer", "further", "in front of", and "behind". In addition, in a manner similar to that described above, the object selection system 106 can select one or more query object instances that satisfy the depth object property.

[0170] Fig.13 A graphical user interface is shown for utilizing an object localization detection model to detect target instances of a query object in accordance with one or more embodiments. Fig.13 The client device 300 described above is included. For example, the client device 300 includes an image editing application that implements the image editing system 104 using the object selection system 106 .

[0171] like Fig.13As shown, graphical user interface 1302 includes an image 1304 within an image editing application. Image 1304 shows two boys in the foreground and trees and mountains in the background. Graphical user interface 1302 also includes object selection request interface 1306, as described above in conjunction with Figure 3A As described, where the user provides the query string "boy on the left".

[0172] As previously described, the object selection system 106 utilizes the object selection pipeline 400 as described above to determine the two boys based on the query object of "boy" in the query string. In addition, as described above, the object selection system 106 can determine the object property of "left" in the query string, which corresponds to the object location property. Therefore, the object selection system 106 selects and utilizes the object location detection model to identify and select the left boy in the image (e.g., the selected target query object 1308). Therefore, as described above, Fig.13 As shown, the object selection system 106 can return the selected target query object 1308 to the user within the image editing application.

[0173] Although the above embodiments involve using an object color detection model or an object location detection model to detect a target query object instance, in alternative embodiments, the object selection system 106 may select and utilize other specialized object attribute detection models and neural networks to detect a target query object instance. For example, based on analyzing adjectives in the query string as described above, the object selection system 106 may determine that the attribute is size, length, shape, material, pattern, position, depth, rigidity / flexibility, protrusion, body posture, emotion, facial expression, or other known attribute types of the query object.

[0174] Based on the selected object attribute type, the object selection system 106 may select a corresponding specialized object attribute detection model. For example, if the object attribute type is material, the object selection system 106 selects an object material detection neural network that identifies a material (e.g., wood, metal, fabric, plastic, ceramic, concrete, glass, a composite material, or another material) associated with each query object instance. As another example, if the object attribute type is shape, the object selection system 106 selects a shape attribute model that identifies the shape of each query object instance. In a similar manner, the object selection system 106 may select various specialized object attribute detection neural networks or models based on the identified object attribute type. Additionally, the object selection system 106 may utilize the selected specialized object attribute detection model and the object attribute to identify the detected target instance of the query object.

[0175] Go to Fig.14 , shows a flow chart of detecting target instances of a query object using a general object attribute detection model according to one or more embodiments. As shown and described above, Fig.14 In the above about Figure 8 The described actions 806 and 810 are expanded, which are themselves an expansion of action 410 of the object selection pipeline 400. Also as shown, Fig.14 Actions 1402 - 1406 are included, which correspond to the object selection system 106 selecting and utilizing a generic object attribute detection neural network.

[0176] As shown, action 806 may include action 1402 in which the object selection system 106 determines that the object attribute type does not correspond to a known object attribute type. For example, based on analyzing the adjectives in the query string as described above, the object selection system 106 may determine that the object attribute is not one of the identified or recognized object attribute types. Therefore, the object selection system 106 may select a generic object attribute detection model, as shown in action 1404.

[0177] As shown, Fig.14 Action 810 in may include action 1406 of the object selection system 106 identifying a target instance of the detected query object using the general object attribute detection model and the object attributes. To illustrate, in one or more embodiments, the general object attribute detection model may generate a label for each query object instance (e.g., using automatic labeling techniques) to associate each instance with various attributes. For example, the general object attribute detection model is a neural network trained to predict labels based on one or more attributes and / or features recognized from the image and assign the label to a given image. In practice, the general object attribute detection model may discern relevant adjective labels and extract the relevant adjective labels from the query object instance bounding box.

[0178] Specifically, the object selection system 106 may provide a bounding box (e.g., a cropped image) of each query object instance to a tagging neural network. The tagging neural network may then generate a list of tags for each query object instance. In one or more embodiments, the tagging neural network includes one of the networks disclosed in U.S. Patent Publication No. 2018 / 0267997 filed by Lin et al. on September 20, 2018, the entire contents of which are incorporated herein by reference. Specifically, the tagging neural network may include an encoder and a decoder. The encoder takes a bounding box (e.g., a query object instance) as input to generate a feature map. The decoder decodes the feature map to identify which tags are associated with the query object instance. In one or more embodiments, the tagging neural network is a classification deep neural network that determines the probability (e.g., confidence) that each tag in a tag set is associated with a query object instance. The tagging neural network may associate a confidence score with each tag. The object selection system 106 may associate tags with query object instances having confidence scores above a threshold.

[0179] After using the label neural network to automatically tag each of the query object instances with the label, object selection system 106 can compare the object attributes provided in the query string with the attribute labels generated for each of the query object instances. To illustrate, in one or more embodiments, object selection system 106 filters out instances that do not have a correlation between the attribute labels and the object attributes or are less than a minimum correlation threshold. In practice, by analyzing the attribute labels of the query object instances with the object attributes from the query string, object selection system 106 can ignore query object instances that do not match the object attributes.

[0180] In additional or alternative embodiments, the object selection system 106 can match the attribute tags of the query object instances with the object attributes from the query string to identify which instances correspond to the object attributes. For example, if the object attribute is "happiness" from the query string of "happy man", the object selection system 106 analyzes the attribute tags of each query object instance to determine whether the tags include "happiness" or a synonym of "happiness".

[0181] After filtering out instances with non-corresponding labels and / or identifying instances with corresponding labels, the object selection system 106 can select specific one or more query object instances corresponding to the object attributes. Therefore, even if a dedicated object attribute detection model is not available for the object attributes, in most cases, the object selection system 106 can still identify the target query object instance from the object attributes using a general object attribute detection neural network.

[0182] Go to Fig.15 , shows a flow chart for detecting and selecting target instances of a query object in an image based on multimodal input according to one or more embodiments. Fig.15 Described is a target query object instance for detecting a query object detected within an image based on additional user input indicating a location in the image. As shown, Fig.15 Included is a series of actions 1500 that the object selection system 106 may perform.

[0183] As shown, the series of actions 1500 may include an action 1502 of the object selection system 106 acquiring an image. For example, a user captures an image using a camera on a client device associated with the user. In addition, the series of actions 1500 may include an action 1504 of the object selection system 106, as previously described, obtaining a query string from the user.

[0184] Additionally, a series of actions 1500 includes an action 1506 in which the object selection system 106 obtains additional user input indicating a location within the image. In one or more embodiments, the additional user input is a touch input selection indicating a specific location within the image. In some embodiments, the additional user input is a mouse (or equivalent) input selection indicating a specific location within the image. The range of the location can range from a single point within the image (e.g., pixel coordinates) to a portion of the image (e.g., a group of pixels). In addition, examples of additional user input include tapping, clicking, sliding, hovering, dragging, scribbling, graffiti, scribbling, or clicking. Other examples of additional user input may include rectangular selection, lasso selection, trace selection, magnetic selection, another type of selection.

[0185] As shown, the series of actions 1500 may include an action 1508 in which the object selection system 106 utilizes the multimodal user input to detect a target query object instance of the detected query object. More specifically, the query string is a first user input, and the location indication is an additional user input. These multiple different user inputs together provide the object selection system 106 with a multimodal user input that enables the object selection system 106 to more accurately detect and select a target query object instance of the detected query object.

[0186] In various embodiments, the query string signals that additional user input includes location information. For example, the object selection system identifies text (e.g., object attributes) in the query string, such as "this" (e.g., "this car") or "here" (e.g., "here cup"), which indicates that the user has provided (or will provide) location input related to the query string. For example, when the object selection system 106 may receive a request to select "this elephant", the object selection system 106 provides an additional prompt for the user to click, tap, outline, doodle, or otherwise indicate the location of the elephant within the picture.

[0187] In addition, the object selection system 106 can utilize the user location input to select a target query object instance and / or filter out other query object instances of the detected query object. For example, using the indicated location from the additional user input, the object selection system 106 can filter out instances of the detected query object that do not match the indicated location or are more than a threshold distance away from the indicated location. As another example, the object selection system 106 can rank or sort multiple query object instances based on their distances to the indicated location, where a higher ranking corresponds to a higher probability of being the target query object instance. Additionally, the object selection system 106 can select an object mask having an instance that is closest to the center of the indicated location. In this way, the object selection system 106 can utilize the user location input in a manner similar to object attributes identified in a query string.

[0188] like Fig.15 As shown, series of actions 1500 may include an action 1510 of object selection system 106 providing the selected query object instance to the user. As described above, object selection system 106 may automatically select a target query object instance within the image editing application in response to a selection query, in which case the selection query includes a multimodal input.

[0189] Reference now Fig.16 , provides additional details about the capabilities and components of the object selection system 106 according to one or more embodiments. Specifically, Fig.16 A schematic diagram showing an example architecture of an object selection system 106 implemented within an image editing system 104 and hosted on a computing device 1600 is shown. The image editing system 104 may correspond to the previously described Figure 1 An image editing system 104 is described.

[0190] As shown, object selection system 106 is located on computing device 1600 within image editing system 104. In general, computing device 1600 can represent various types of client devices. For example, in some embodiments, the client is a mobile device, such as a laptop, tablet computer, mobile phone, smart phone, etc. In other embodiments, computing device 1600 is a non-mobile device, such as a desktop or server, or another type of client device. Additional details about computing device 1600 are provided below and with respect to Fig.18 discuss.

[0191] like Fig.16 As shown, the object selection system 106 includes various components for performing the processes and features described herein. For example, the object selection system 106 includes a digital image manager 1610, a user input detector 1612, an object detection neural network manager 1614, an object mask generator 1616, an object instance selector 1618, and a storage manager 1620. As shown, the storage manager 1620 includes a digital image 1622, an object detection neural network 1624, an object mask neural network 1626, and an object attribute detection model 1628. Each of the above components is described in turn below.

[0192] As described above, the object selection system 106 includes a digital image manager 1610. In general, the digital image manager 1610 facilitates identifying, accessing, receiving, obtaining, generating, importing, exporting, copying, modifying, removing, and organizing images. In one or more embodiments, the digital image manager 1610 operates in conjunction with the image editing system 104 (e.g., an image editing application) to access and edit images, as previously described. In some embodiments, the digital image manager 1610 communicates with a storage manager 1620 to store and retrieve digital images 1622, for example, within a digital image database managed by the storage manager 1620.

[0193] As shown, the object selection system 106 includes a user input detector 1612. In various embodiments, the user input detector 1612 can detect, receive, and / or facilitate user input on the computing device 1600 in any suitable manner. In some cases, the user input detector 1612 detects one or more user interactions (e.g., a single interaction or a combination of interactions) with respect to a user interface. For example, the user input detector 1612 detects user interactions from a keyboard, a mouse, a touch page, a touch screen, and / or any other input device associated with the computing device 1600. For example, the user input detector 1612 detects user input of a query string submitted from an object selection request interface, which user input requests automatic selection of an object within an image. Additionally, as described above, the user input detector 1612 detects additional user input from a mouse selection and / or touch input to indicate an object location within an image.

[0194] As shown, the object selection system 106 includes an object detection neural network manager 1614. In various embodiments, the object detection neural network manager 1614 maintains, creates, generates, trains, updates, accesses, and / or utilizes the object detection neural network disclosed herein. As described above, the object detection neural network manager 1614 detects one or more objects (e.g., a query object) within an image and generates an approximate boundary (e.g., a bounding box) to indicate the detected objects.

[0195] Additionally, in various embodiments, object detection neural network manager 1614 may communicate with storage manager 1620 to store, access, and utilize object detection neural network 1624. As described above, in various embodiments, object detection neural network 1624 may include one or more specialized object detection neural networks, object-based concept detection neural networks, known object category detection neural networks, unknown object category detection neural networks, object proposal neural networks, region proposal neural networks, concept embedding neural networks.

[0196] Additionally, as shown, the object selection system 106 includes an object mask generator 1616. In one or more embodiments, the object mask generator 1616 generates, creates, and / or produces an accurate object mask based on the detected object. For example, the object detection neural network manager 1614 provides an approximate boundary of an object (e.g., a detected query object) to the object mask generator 1616, and the object mask generator 1616 generates an object mask for the detected object using the object mask neural network 1626, as described above. Also as described above, in various embodiments, when multiple instances of the query object are detected, the object mask generator 1616 generates multiple object masks.

[0197] As shown, the object selection system 106 includes an object instance selector 1618. In some embodiments, the object instance selector 1618 determines, analyzes, detects, identifies, filters, and / or selects one or more detected target instances from a plurality of detected instances of an object. In various embodiments, as described above, the object instance selector 1618 utilizes one or more object attribute detection models 1628 to identify the target instance. For example, the object instance selector 1618 utilizes object attributes to select a target instance for a plurality of detected instances of an object.

[0198] Examples of object attribute detection models 1628 include, but are not limited to, object color detection model 1630, object location detection model 1632, general object attribute detection neural network 1634, and / or other object attribute detection neural networks and models (e.g., object depth detection neural network, object material detection neural network, object shape detection model, object size detection neural network, object length detection neural network, object pattern detection neural network, object location detection model, object body posture detection neural network, object salience detection neural network, object facial expression detection neural network, or object quality detection neural network).

[0199] Each of the components 1610-1634 of the object selection system 106 may include software, hardware, or both. For example, the components 1610-1634 may include one or more instructions, one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as a client device (e.g., a mobile client device) or a server device. When executed by one or more processors, the computer-executable instructions of the object selection system 106 may cause the computing device to perform the feature learning method described herein. Alternatively, the components 1610-1634 may include hardware, such as a dedicated processing device for performing a specific function or group of functions. Additionally, the components 1610-1634 of the object selection system 106 may include a combination of computer-executable instructions and hardware.

[0200] In addition, the components 1610-1634 of the object selection system 106 can be implemented as one or more operating systems, one or more stand-alone applications, one or more modules of an application, one or more plug-ins, one or more library functions or other functions that can be called by other applications, and / or a cloud computing model. Therefore, the components 1610-1634 can be implemented as stand-alone applications, such as desktop or mobile applications. In addition, the components 1610-1634 can be implemented as one or more web-based applications hosted on a remote server. The components 1610-1634 can also be implemented in a set of mobile device applications or "apps". For illustration, the components 1610-1634 can be implemented in applications, including but not limited to

[0201] “ADOBE”, “INDESIGN”, “ACROBAT”, “ILLUSTRATOR”, “PHOTOSHOP” and “CREATIVE CLOUD” are registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries.

[0202] Figures 1 to 16 , the corresponding text and examples provide many different methods, systems, devices and non-transitory computer-readable media of the object selection system 106. In addition to the foregoing, one or more embodiments may also be described in terms of flowcharts including actions for achieving specific results, such as Fig.17 Additionally, the actions described herein may be repeated or performed in parallel with each other or with different instances of the same or similar actions.

[0203] As mentioned, Fig.17 1700 is a flow chart showing a series of actions 1700 for detecting a query object instance using one of a plurality of object property detection models in accordance with one or more embodiments. Fig.17 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify Fig.17 Any of the actions shown. Fig.17 The actions of may be performed as part of a method. Alternatively, a non-transitory computer readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform Fig.17 In some embodiments, the system may perform Fig.17 action.

[0204] In one or more embodiments, the series of actions 1700 is implemented on one or more computing devices, such as the client device 102, the server device 110, the client device 300, or the computing device 1600. In addition, in some embodiments, the series of actions 1700 is implemented in a digital environment for creating or editing digital content (e.g., a digital image). For example, the series of actions 1700 is implemented on a computing device having a memory, the memory including: a digital image; a query, the query including a query object and an object attribute corresponding to the query object (e.g., an object color attribute or an object positioning attribute); and a trained object detection neural network.

[0205] The series of actions 1700 includes an action 1710 of identifying a query object and an object attribute. Specifically, action 1710 may include identifying a query object and a corresponding object attribute from a selection query corresponding to a digital image. In various embodiments, action 1710 may include identifying the query object as a noun in the query (e.g., a query string), and identifying the object attribute as an adjective in the query.

[0206] As shown, a series of actions 1700 also includes an action 1720 of detecting multiple instances of a query attribute. Specifically, action 1720 may involve utilizing an object detection model to detect multiple instances of a query object in a digital image. In one or more embodiments, action 1720 may include identifying a separate bounding box within the digital image for each of the multiple query object instances. In various embodiments, action 1720 may include identifying a separate object mask within the digital image for each of the multiple query object instances. In some embodiments, the object detection model is a trained object detection neural network. In an alternative embodiment, the object detection model is not a machine learning model, such as a combination of a region proposal detector or another type of object detector.

[0207] like Fig.17 As shown, the series of actions 1700 also includes an action 1730 of determining an object property detection model based on the query object. Specifically, action 1730 may include determining an object property detection model from a plurality of object property detection models based on analyzing the object property. In one or more embodiments, action 1730 may include: based on determining that the object property corresponds to a material, selecting an object material detection neural network as the determined object property detection model. In some embodiments, action 1730 may include selecting an object size detection model as the determined object property detection model based on determining that the object property corresponds to an object size.

[0208] As shown, the series of actions 1700 also includes an action 1740 of detecting a query object instance based on the object property detection model. Specifically, action 1740 may include detecting that a first query object instance is a target query object instance from a plurality of query object instances based on the object property and the determined object property detection model. Action 1740 may include many additional actions and sub-actions, as described below.

[0209] In one or more embodiments, act 1740 may include analyzing the object properties to identify the indicated location within the digital image. In these embodiments, act 1740 may also include determining the location within the digital image for each of the plurality of query object instances using the object location property detection model. Act 17440 may also involve detecting that the first query object instance is the target query object instance based on the first query object instance having a location in the digital image that is closest to the indicated location.

[0210] In various embodiments, act 1740 may include analyzing the object properties to identify the indicated relative object location. In these embodiments, act 1740 may also include utilizing the object location property detection model to identify a center location for each of a plurality of query object instances within the digital image, comparing a first center location of a first query object instance of the plurality of query object instances with a second center location of a second query object instance to identify a first relative location of the first query object instance, and detecting that the first query object instance is a target query object instance based on the first relative location of the first query object instance corresponding to the indicated relative object location.

[0211] In some embodiments, action 1740 may include analyzing the object properties to identify the indicated object depth. In these embodiments, action 1740 may also include: generating a depth map of the image using a depth map neural network; based on the depth map, identifying a first depth of the first query object instance using the depth map neural network; and detecting that the first query object instance is a target query object instance based on the first depth of the first query object instance corresponding to the indicated object depth.

[0212] In one or more embodiments, act 1740 may include analyzing the object properties to identify the indicated color. In these embodiments, act 1740 may also include identifying the color of each of a plurality of query object instances in the digital image using an object color property detection model, and determining that a first query object instance has a greater correspondence with the indicated color than other instances of the plurality of query object instances based on comparing the colors identified for each of the plurality of query objects to the indicated color. In additional embodiments, act 1740 may include identifying the color of the first query object instance using an object color property detection model based on comparing one or more pixels of the first query object instance to the indicated color in a multidimensional color space.

[0213] In various embodiments, action 1740 may include determining, based on analyzing the object property, that the object property does not correspond to a known object property type. In these embodiments, action 1740 may also include: generating a label for one or more of the plurality of query object instances using a label neural network; and detecting the first query object instance as a target query object instance based on matching the object property with the label generated for the first query object instance. In additional embodiments, action 1740 may include filtering out one or more other query object instances of the plurality of query object instances based on one or more other instances of the plurality of query object instances having a label that does not correspond to the object property.

[0214] like Fig.17 As shown, the series of actions 1700 also includes an action 1750 of providing the selected query object instance within the image. Specifically, action 1750 can involve providing the detected first query object instance to the digital image in response to the selection query. In some embodiments, action 1750 includes automatically selecting the detected query object within the image editing application using the first object mask.

[0215] In one or more embodiments, act 1750 may include generating a first object mask for the first query object instance using the object mask neural network without generating additional object masks for other query object instances of the plurality of query object instances. In additional embodiments, act 1750 may include selecting the first query object within the image using the first object mask.

[0216] In some embodiments, actions 1730 and 1740 may also include the following actions: determining that the object property corresponds to a color; identifying one or more colors for each of a plurality of query object instances in the digital image; and determining, using the object color detection model, that a first query object instance from the plurality of query object instances includes a first color that corresponds to the object color property in the query. In these embodiments, action 1740 may also include: generating a separate object mask for each of the plurality of query object instances before identifying the one or more colors for each of the plurality of query object instances; and identifying the first color of the first query object instance based on analyzing that one or more pixels of the first object mask correspond to the first query object instance.

[0217] Additionally, in one or more embodiments, determining that the first query object instance includes a first color corresponding to the object color attribute in the query is based on determining that a threshold number of pixels in the first object mask correspond to the object color attribute in the query (e.g., valid pixels). In some embodiments, determining that the first color of the first query object instance corresponds to the object color attribute in the query is based on generating a first color vector for the first color of the first query object instance using an object color detection model, generating an object color attribute vector for the object color attribute of the query using the object color detection model, and determining that the first color vector corresponds to the object color attribute vector using a vector space generated by the object color detection model.

[0218] In various embodiments, acts 1730 and 1740 may include determining that the object property is an object location property indicating a location of a query object instance within the image, identifying a location of each of a plurality of query object instances in the digital image, and utilizing object location detection to determine that a first query object instance from the plurality of query object instances has a first position corresponding to the object location property in the query. In one or more embodiments, act 1740 may also include generating a bounding box for each of the plurality of query object instances.

[0219] Furthermore, in these embodiments, identifying the location of each of the plurality of query object instances in the digital image is based on a bounding box generated for each of the plurality of query object instances. In additional embodiments, act 1750 may include generating a first object mask for a first query object instance using an object mask neural network (e.g., based on determining that the first query object instance corresponds to an object location attribute) without generating additional object masks for other query object instances of the plurality of query object instances.

[0220] As used herein, the term "digital environment" generally refers to an environment implemented, for example, as a standalone application (e.g., a personal computer or mobile application running on a computing device), as an element of an application, as a plug-in to an application, as a library function, as a computing device, and / or as a cloud computing system. As described herein, the digital media environment allows an object selection system to create, execute, and / or modify an object selection pipeline.

[0221] Embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Specifically, one or more processes described herein may be implemented at least in part as instructions that are embodied in a non-transient computer-readable medium and are executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transient computer-readable medium (e.g., a memory) and executes those instructions to perform one or more processes, including one or more processes described herein.

[0222] Computer-readable media can be any available media that can be accessed by a general or special purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transient computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct types of computer-readable media: a non-transient computer-readable storage medium (device) and a transmission medium.

[0223] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase change memory ("PCM"), other types of memory, other optical disk devices, magnetic disk devices or other magnetic storage devices, or any other media that can be used to store the desired program code in the form of computer-executable instructions or data structures and can be accessed by a general or special purpose computer.

[0224] A "network" is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer correctly views the connection as a transmission medium. The transmission medium may include a network and / or data links that may be used to carry the desired program code components in the form of computer executable instructions or data structures and that may be accessed by a general or special purpose computer. Combinations of the above should also be included within the scope of computer readable media.

[0225] In addition, upon reaching various computer system components, program code components in the form of computer executable instructions or data structures can be automatically transferred from the transmission medium to the non-transient computer readable storage medium (device) (or vice versa). For example, computer executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., "NIC"), and then ultimately transferred to the computer system RAM and / or a less volatile storage medium (device) at the computer system. Therefore, it should be understood that non-transient computer readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0226] Computer executable instructions include, for example, instructions and data, which, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or function group. In some embodiments, computer executable instructions are executed by a general-purpose computer to transform a general-purpose computer into a special-purpose computer that implements an element of the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in a language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned features or actions. On the contrary, the described features and actions are disclosed as example forms that implement the claims.

[0227] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment, in which both local and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or by a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can be located in local and remote storage devices.

[0228] Embodiments of the present disclosure may also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers to a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in the marketplace to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be rapidly provisioned via virtualization, released with less management effort or service provider interaction, and then scaled accordingly.

[0229] The cloud computing model may consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, and the like. The cloud computing model may also expose various service models, such as, for example, software as a service (“SaaS”), platform as a service (“PaaS”), and infrastructure as a service (“IaaS”). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, and the like. Additionally, as used herein, the term “cloud computing environment” refers to an environment in which cloud computing is employed.

[0230] Fig.18A block diagram of an example computing device 1800 is shown, which can be configured to perform one or more of the above processes. It will be understood that one or more computing devices such as computing device 1800 can represent the above computing devices (e.g., client device 102, server device 110, client device 300, or computing device 1600). In one or more embodiments, computing device 1800 can be a mobile device (e.g., a laptop computer, a tablet computer, a smart phone, a mobile phone, a camera, a tracker, a watch, a wearable device, etc.). In some embodiments, computing device 1800 can be a non-mobile device (e.g., a desktop computer, a server device, a web server, a file server, a social networking system, a program server, an application store, or a content provider). In addition, computing device 1800 can be a server device including cloud-based processing and storage capabilities.

[0231] like Fig.18 As shown, computing device 1800 may include one or more processors 1802, memory 1804, storage device 1806, input / output ("I / O") interface 1808, and communication interface 1810, which may be communicatively coupled by way of a communication infrastructure (e.g., bus 1812). Fig.18 1800 is shown, but Fig.18 The components shown in FIG. 1 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in some embodiments, computing device 1800 includes more than Fig.18 The components shown are fewer than the components shown. Fig.18 Components of computing device 1800 are shown.

[0232] In certain embodiments, processor(s) 1802 include hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, to execute an instruction, processor(s) 1802 may retrieve (or fetch) the instruction from an internal register, an internal cache, memory 1804, or storage device 1806, and decode and execute the instruction.

[0233] Computing device 1800 includes memory 1804 coupled to processor(s) 1802. Memory 1804 may be used to store data, metadata, and programs executed by processor(s). Memory 1804 may include one or more of volatile and non-volatile memory, such as random access memory ("RAM"), read-only memory ("ROM"), solid-state disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. Memory 1804 may be internal or distributed memory.

[0234] The computing device 1800 includes a storage device 1806, which includes a storage device for storing data or instructions. As an example, but not by way of limitation, the storage device 1806 may include the non-transitory storage medium described above. The storage device 1806 may include a hard disk drive (HDD), a flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0235] As shown, computing device 1800 includes one or more I / O interfaces 1808, which are provided to allow a user to provide input (e.g., user strokes) to computing device 1800, receive output from computing device 1800, and otherwise transfer data to and from computing device 1800. These I / O interfaces 1808 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of these I / O interfaces 1808. The touch screen may be activated with a stylus or a finger.

[0236] I / O interface 1808 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O interface 1808 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0237] The computing device 1800 may further include a communication interface 1810. The communication interface 1810 may include hardware, software, or both. The communication interface 1810 provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example and not limitation, the communication interface 1810 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network such as WI-FI. The computing device 1800 may further include a bus 1812. The bus 1812 may include hardware, software, or both that connect the components of the computing device 1800 to each other.

[0238] In the foregoing description, the present invention has been described with reference to specific exemplary embodiments of the present invention. Various embodiments and aspects of the invention(s) are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are illustrative of the present invention and should not be construed as limiting the present invention. Many specific details are described to provide a thorough understanding of the various embodiments of the present invention.

[0239] Without departing from the spirit or essential characteristics of the present invention, the present invention may be implemented in other specific forms. The described embodiments are considered to be illustrative and non-restrictive in all respects. For example, the method described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in different orders. Additionally, the steps / actions described herein may be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes falling within the equivalent meanings and scope of the claims will be included within their scope.

Claims

1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to: receiving, based on identifying a digital image including a plurality of detectable objects, a user input from a client device, the user input including a selection query indicating a target query object to be selected within the digital image; analyzing the select query to identify a first detection input including a query object and a second detection input including an object attribute corresponding to the query object; detecting a plurality of query object instances in the digital image from the plurality of detectable objects using an object detection neural network based on the first detection input; selecting, based on the second detection input, an object attribute detection model specifically trained to identify the object attribute indicated by the selection query from a plurality of object attribute detection models designed to identify corresponding object attributes; By using the object property detection model to determine that a first query object instance from the plurality of query object instances reflects the object property, detecting that the first query object instance is a target query object is performed by: identifying which of the object masks includes the object attribute by analyzing the object masks of the plurality of query object instances, determining that the first relative position of the first query object instance corresponds to the object attribute by identifying the first relative position of the first query object instance based on comparing the first central position of the first query object instance with the second central position of a second query object instance among the plurality of query object instances; as well as A digital image having the target query object selected in response to the select query is provided to the client device.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to parse the select query from the client device to determine: a noun identifying the query object; and An adjective that identifies a property of the object.

3. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: determining a location of an indication of the query object within the digital image based on the object attributes of the selection query; and The first query object instance is detected to be the target query object by performing the following operations: utilizing an object location attribute detection model to determine a location within the digital image for each of the plurality of query object instances; and The first query object instance is detected to be the target query object based on the first query object instance having a location in the digital image that is closest to the indicated location.

4. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: determining a relative object location of an indication of the query object within the digital image based on the object attributes of the selection query; and Based on the first relative position for the first query object instance corresponding to the indicated relative object position, detecting that the first query object instance is the target query object.

5. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: determining an indicated object depth of the query object within the digital image based on the object attribute of the selected query; and Detecting that the first query object instance is the target query object by: generating a depth map of the digital image using a deep map neural network; identifying, using the deep map neural network, a first localized depth of the first query object instance based on the depth map; and Based on the first located depth of the first query object instance corresponding to the indicated object depth, the first query object instance is detected to be the target query object.

6. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: Determining, based on the object attribute of the selected query, an indicative facial expression of the query object in the digital image; By determining that the first query object instance reflects the indicated facial expression, it is detected that the first query object instance is the target query object.

7. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: determining an object shape of the query object within the digital image based on the object attribute of the selected query; and By identifying the first query object instance corresponding to the object shape, it is detected that the first query object instance is the target query object.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: Determining the indicated color of the query object in the digital image according to the object attribute of the selection query; and The first query object instance is detected to be the target query object by performing the following operations: identifying a color for each of the plurality of query object instances in the digital image using an object color attribute detection model; and Based on comparing the color identified for each of the plurality of query object instances with the indicated color, it is determined that the first query object instance has a greater correspondence with the indicated color than other instances of the plurality of query object instances.

9. The non-volatile computer-readable medium of claim 8, further comprising instructions which, when executed by the at least one processor, cause the computing device to: identify the color for the first query object instance using the object color attribute detection model by comparing one or more pixels of the first query object instance with the indicated color in a multidimensional color space.

10. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: determining that the object attribute does not correspond to a known object attribute type; and The first query object instance is detected to be the target query object based on determining that the object attribute does not correspond to a known object attribute type by: generating a label for one or more query object instances among the plurality of query object instances using a label neural network; and The object attributes are matched with tags generated for the first query object instance.

11. The non-volatile computer-readable medium of claim 10, wherein the instructions, when executed by the at least one processor, cause the computing device to detect that the first query object instance is the target query object by filtering out one or more other query object instances from the plurality of query object instances based on the one or more other query object instances having a label that does not correspond to the object attribute.

12. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: Determining the object material indicated by the query object in the digital image according to the object attribute of the selected query; By identifying the first query object instance corresponding to the indicated object material using an object material detection model, it is detected that the first query object instance is the target query object.

13. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: Determining the indicated object size of the query object in the digital image according to the object attribute of the selection query; and By identifying the first query object instance corresponding to the indicated object size using an object size detection model, it is detected that the first query object instance is the target query object.

14. A system for automatically selecting an object within a digital image, comprising: Memory devices, including: Digital images; A selection query indicates a target query object to be selected within the digital image by including a first detection input indicating a query object and a second detection input indicating an object attribute corresponding to the query object; and Object detection neural networks; and at least one processor configured to cause the system to: detecting a plurality of query object instances of the query object in the digital image using the object detection neural network and the first detection input based on identifying the first detection input indicative of the query object within the selection query; selecting, based on the second detection input indicating the object attribute from the select query, an object attribute detection model specifically trained to identify the object attribute indicated by the select query from a plurality of object attribute detection models designed to identify respective object attributes; By using the object property detection model to determine that a first query object instance from the plurality of query object instances reflects the object property, detecting that the first query object instance is the target query object is performed by: identifying which of the object masks includes the object attribute by analyzing the object masks of the plurality of query object instances, determining that the first relative position of the first query object instance corresponds to the object attribute by identifying a first relative position of the first query object instance based on comparing a first central position of the first query object instance with a second central position of a second query object instance among the plurality of query object instances; and A digital image having the first query object instance selected in response to the selection query is provided for digital editing within the digital image.

15. The system of claim 14, wherein the at least one processor is configured to cause the system to detect a plurality of query object instances in the digital image by: generating approximate boundaries for the plurality of query object instances using the object detection neural network; and An object mask for each of the plurality of query object instances is generated from the approximate boundaries using an object mask neural network.

16. The system of claim 15, wherein the at least one processor is further configured to cause the system to: determining additional object properties of the query object in the digital image based on a third detection input of the select query; and By determining that the first query object instance reflects the object attribute and reflects the additional object attribute using an object attribute detection model and an additional attribute detection model, it is detected that the first query object instance is the target query object.

17. The system of claim 16, wherein the at least one processor is configured to cause the system to: determine that pixels in a first object mask more closely correspond to a first color than pixels in object masks for other query object instances in the plurality of query object instances.

18. A computer-implemented method of selecting a query object in a digital media environment for creating or editing digital images, comprising: receiving, based on identifying a digital image including a plurality of detectable objects, a user input from a client device, the user input including a selection query indicating a target query object to be selected within the digital image; analyzing the select query to identify a first detection input including a query object and a second detection input including an object attribute corresponding to the query object; detecting, using an object detection neural network, a plurality of query object instances of the query object in the digital image from the plurality of detectable objects based on the first detection input; selecting, based on the second detection input, an object attribute detection model specifically trained to identify the object attribute from a plurality of object attribute detection models designed to identify corresponding object attributes; By using the object property detection model to determine that a first query object instance from the plurality of query object instances reflects the object property, detecting that the first query object instance is the target query object is performed by: identifying which of the object masks includes the object attribute by analyzing the object masks of the plurality of query object instances, determining that the first relative position of the first query object instance corresponds to the object attribute by identifying a first relative position of the first query object instance based on comparing a first central position of the first query object instance with a second central position of a second query object instance among the plurality of query object instances; as well as The digital image with the first query object instance selected in response to the selection query is provided to the client device for editing within the digital image.

19. The computer-implemented method of claim 18, further comprising: generating a bounding box for each query object instance in the plurality of query object instances; and Wherein identifying the location of each query object instance in the plurality of query object instances in the digital image is based on the bounding box generated for each query object instance in the plurality of query object instances.

20. The computer-implemented method of claim 19, further comprising: generating a first object mask for the first query object instance using an object mask neural network without generating additional object masks for other query object instances in the plurality of query object instances; as well as The first query object instance is selected within the image using the first object mask.

Images