Classify the color of objects in digital images

Abstract

Embodiments of the present disclosure relate to classifying the color of objects in digital images. This disclosure relates to a color classification system that accurately classifies objects in digital images based on color. Specifically, in one or more embodiments, the color classification system utilizes a multidimensional color space and one or more color maps to match objects to colors. In practice, the color classification system can utilize one or more color similarity regions generated in the multidimensional color space to accurately and efficiently detect the color of an object.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to digital images, and more particularly to classifying the color of objects in digital images. Background Art

[0002] In recent years, there has been a rapid increase in the use of digital images. Indeed, advances in both hardware and software have enhanced the ability of individuals to capture, create, edit, search, and share digital images. For example, the hardware on most modern computing devices (e.g., servers, desktop computers, laptops, tablet computers, and smartphones) supports digital image editing and sharing without significant processing delays. Similarly, improvements in software have enabled individuals to modify, search, share, or otherwise utilize digital images.

[0003] With the increasing use of digital images, there is an increasing demand for systems that can quickly and efficiently classify and detect digital images, especially identifying objects in images based on their color. However, conventional image selection systems have many problems in terms of operational flexibility, accuracy, and efficiency.

[0004] As an example of inaccuracy, many conventional systems are unable to accurately classify objects associated with a given color in an image. In particular, conventional systems employ simple color classification schemes that fail to capture the complexity and nuances of color matching associated with human perception of color. Exacerbating this problem, conventional systems often fail to identify objects in an image as having a given color when the objects in the image are visually affected due to poor capture conditions (e.g., lighting, angle, exposure, or clarity) or other image variations.

[0005] In some cases, conventional systems inaccurately misclassify objects as a given color when they are not actually that color. Again, conventional systems produce false positives because they fail to accurately account for variations in image quality and conditions. Consequently, individuals are left to manually remove images that were incorrectly identified or classified.

[0006] Furthermore, conventional systems are inefficient. For example, conventional systems require significant processing resources to classify and detect objects of a given color within an image. As image size continues to increase, the amount of computer resources and memory required to classify and detect objects in an image will increase significantly. Furthermore, as mentioned above, when conventional systems misclassify and / or misidentify objects, individuals must correct these errors, which requires significant time and user interaction. Furthermore, conventional systems waste significant computing resources when misclassifying and misidentifying objects in an image.

[0007] Furthermore, conventional systems have significant drawbacks related to operational flexibility. For example, conventional systems cannot handle differences in color properties when detecting objects associated with a given color. Furthermore, the rigidity of conventional systems prevents them from operating in a manner that flexibly captures human perception of color matching.

[0008] These and additional problems related to classifying and detecting objects and object colors in digital images exist in image selection systems. Summary of the Invention

[0009] Embodiments of the present disclosure utilize systems, non-transitory computer-readable media, and methods for classifying objects based on color in digital images to provide benefits and / or address one or more of the foregoing or other problems in the art. For example, the disclosed system can utilize an improved color classifier model that can accurately and efficiently classify the color of objects in an image.

[0010] For illustration, the disclosed system can identify a color similarity region for a color (e.g., a target color) or separate color similarity regions for multiple colors within a multidimensional color space. Additionally, the disclosed system can identify objects in a digital image. For pixels representing an object, the disclosed system can map the pixels to the multidimensional color space to determine color correspondence with a color similarity region (or multiple color similarity regions for multiple colors). Additionally, the disclosed system can generate a color match score for the object based on the color correspondence between the object pixels and the color similarity regions. Furthermore, the disclosed system can classify the object as the color based on the color match score (e.g., satisfying a minimum color match threshold for the color).

[0011] Although for simplicity, this summary refers to the disclosed system, this summary also applies to certain disclosed methods and non-transitory computer-readable media. The following description sets forth additional features and advantages of one or more embodiments of the disclosed systems, computer media, and methods. In some cases, such features and advantages will be apparent to those skilled in the art or may be learned through practice of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] As briefly described below, the detailed description provides additional specificity and detail to one or more embodiments through the use of accompanying figures.

[0013] Figure 1 A schematic diagram illustrating an environment in which a color classification system may operate in accordance with one or more embodiments is illustrated.

[0014] Figure 2Illustrated is a flow diagram for classifying a query object within a digital image into a target color in accordance with one or more embodiments.

[0015] Figure 3A-Figure 3B Illustrated is mapping of a query color to a color space in accordance with one or more embodiments.

[0016] Figures 4A-4C Illustrated is the generation and mapping of candidate query colors to a color space in an image in accordance with one or more embodiments.

[0017] Figures 5A-5E Illustrated is the generation of a color matching score for a query object in accordance with one or more embodiments.

[0018] Figure 6 Illustrated is a flowchart for filtering query object instances based on color matching scores in accordance with one or more embodiments.

[0019] Figures 7A-7C Illustrated is a graphical user interface for identifying digital images that include query objects that match a query color in accordance with one or more embodiments.

[0020] Figure 8A-8B Illustrated is a graphical user interface for detecting query objects in a digital image that match a query color in accordance with one or more embodiments.

[0021] Figures 9A-9D Illustrated is a graphical user interface for utilizing an object detection network to detect multiple query object instances in an image in accordance with one or more embodiments.

[0022] Figure 10 A diagram illustrating an example architecture of a color classification system in accordance with one or more embodiments is shown.

[0023] Figure 11 Illustrated is a flow diagram illustrating a series of acts for classifying the color of an object in a digital image in accordance with one or more embodiments.

[0024] Figure 12 A flow diagram illustrating a series of actions for detecting a query object corresponding to a query color in a digital image in accordance with one or more embodiments is shown.

[0025] Figure 13 A block diagram of an example computing device for implementing one or more embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION

[0026] This disclosure describes one or more embodiments of a color classification system that accurately classifies the colors of objects in digital images. In particular, in one or more embodiments, the color classification system utilizes a multidimensional color space and one or more color maps to classify the colors of objects. Using a color classifier, the color classification system can accurately and efficiently detect specific instances of objects in an image based on a target color, and identify one or more digital images that include objects that match the target color.

[0027] To illustrate, in one or more embodiments, a color classification system can identify color similarity regions for a color or separate color similarity regions for multiple colors within a multidimensional color space. Additionally, the color classification system can identify objects in a digital image comprised of pixels. The color classification system can map the pixels to the multidimensional color space to determine color correspondences with one or more color similarity regions. Additionally, the color classification system can generate one or more color matching scores for the object for one or more colors based on the color correspondences. Furthermore, the color classification system can classify the object into colors based on the color matching scores.

[0028] As mentioned above, the color classification system can utilize color similarity regions to map the pixels of an object to colors. In various embodiments, the color classification system can generate a color similarity region for a color by mapping the color to a multidimensional color space and mapping alternative / similar colors of the color to the same multidimensional color space. For example, in one or more embodiments, for a target color, the color classification system can map the alternative colors to a color space near the target color. In some embodiments, the color classification system modifies the properties of the target color in the color classification system to generate the alternative colors, as described below.

[0029] As mentioned above, the color classification system can generate candidate colors using multiple color spaces (corresponding to color models). For example, the color classification system can map a given color (e.g., a target color) from a first color model to a second color model, where the color models are defined by different color attributes and / or parameters. In addition, in some embodiments, the color classification system can modify one or more attributes of the given color in the second color model to generate the candidate color in the second color model. The color classification system can then convert the candidate color back to the first color model and map it to a multidimensional color space near the given color.

[0030] A color classification system can maintain color similarity regions for a set of colors. For example, the color classification system selects a list of colors and generates one or more color similarity regions for each color. Then, when classifying an object in an image as a color, the color classification system maps the object's pixels into a multidimensional color space and determines a color correlation between the mapped object pixels and each pre-mapped color similarity region. The color classification system can classify the object as the color whose color similarity region includes the most favorable color correlation (e.g., the most mapped object pixels). In this way, the color classification system can classify the object as a common color and / or a color for which the color classification system has previously generated a color similarity region in the multidimensional color space.

[0031] In some embodiments, the color classification system can be combined with performing object detection to generate a color similarity region for a target color. To illustrate, in one or more embodiments, the color classification system can identify a search query that includes a query color (i.e., a target color) and a corresponding query object. In response to the query, the color classification system can map the query color to a first point in a multidimensional color space, and generate and map alternative query colors to the same color space to form a color similarity region for the query color. In addition, the color classification system can detect the query object in a digital image. Moreover, based on comparing pixels corresponding to the query object with mapped points in the color space corresponding to the query color, the color classification system can generate a color match score for the query object. In addition, the color classification system can provide a digital image to a user based on the query object meeting a color match threshold.

[0032] As mentioned above, the color classification system can classify and detect query objects of a query color in a digital image (or simply, "image") in response to a query. For example, the query can correspond to a search query to identify images that include the query object having the query color. In another example, the query can correspond to a selection query to automatically select one or more instances of the query object in the image having a corresponding query color from multiple instances of the query object in the image.

[0033] As mentioned above, a color classification system can detect a query object in an image. For example, in one or more embodiments, the color classification system can utilize an object detection neural network to detect a query object (of any color) in an image. In some cases, the object detection neural network detects multiple instances of the query object in an image. In various embodiments, the object detection neural network detects the query object in multiple images. Depending on how the object detection neural network is trained, the object detection neural network can detect objects from a variety of object categories and types.

[0034] As mentioned above, in various embodiments, the color classification system can generate a color matching score for a detected query object. For example, in many embodiments, the color classification system can compare the pixels of the object to one or more colors. For example, the color classification system can map each pixel to a multidimensional color space to determine whether the pixel is near a given color or an alternative color to the given color (e.g., within a color similarity region of the color). The color classification system can then calculate the color matching score for the object based on the number or percentage of pixels within the color similarity region (i.e., valid pixels) and pixels outside the color similarity region (i.e., invalid pixels).

[0035] In various embodiments, the color classification system can reduce (i.e., shrink) the pixels of an object. For example, the color classification system can reduce the number of pixels of a query object using one or more of the methods described below. By reducing the number of pixels that are analyzed, mapped, and compared in a multidimensional color space, the color classification system can reduce processing and memory requirements.

[0036] In some embodiments, the color classification system can apply a filtering threshold to the color matching scores of one or more objects. For example, the color classification system can filter out objects whose color matching scores are below a threshold amount. Furthermore, in embodiments where multiple objects (or multiple object instances) are detected, the color classification system can further filter out detected objects whose color matching scores are significantly lower than the color matching scores of other detected objects.

[0037] As mentioned above, color classification systems offer numerous advantages, benefits, and practical applications compared to conventional systems. In particular, color classification systems provide operational flexibility, improved accuracy, and increased efficiency. Specifically, as mentioned above, color classification systems can classify the colors of objects based on mapping a color and one or more alternative colors into a multidimensional color space to form color similarity regions. In many embodiments, the color similarity regions closely mimic human color perception and behavior in terms of color matching accuracy. In this manner, color classification systems significantly improve color matching accuracy compared to conventional systems.

[0038] Additionally, the color classification system accounts for image variations and poor capture conditions by representing colors in a multidimensional color space as similarity regions. By generating and utilizing similarity regions, the color classification system can match objects to their colors, where conventional systems would otherwise fail to find identical color matches. Furthermore, as a result of providing improved accuracy, the color classification system better classifies and detects objects in an image that match the target color. Furthermore, this increased accuracy reduces the number of misclassifications and false positives.

[0039] Furthermore, the color classification system provides improved efficiency compared to conventional systems. As mentioned above, in various embodiments, the color classification system can downsample the pixels on an object before generating a color match score for the object. By reducing the number of pixels per object and / or per object instance, the color classification system reduces the processing and memory resources required to match colors.

[0040] Furthermore, as mentioned above, the improved accuracy of the color classification system results in improved efficiency. In fact, the improved accuracy results in fewer overall computer calculations, with fewer object searches, object detections, pixel color comparisons, object color score calculations, and calculations in response to user interactions. In some embodiments, the color classification system further improves efficiency by stopping color match score calculations when a minimum number / percentage of pixels in an object meets a minimum color match threshold.

[0041] Furthermore, the color classification system provides increased flexibility compared to conventional systems. As mentioned above, in various embodiments, the color classification system can utilize multiple color models rather than a single color model. For example, the color classification system can select a second color model to modify a color attribute that is more refined in a second color space (corresponding to the second color model) than in the first color space. In fact, in some cases, the color classification system will not be able to accurately modify a specific color attribute of a color in only the first color space. Therefore, considering the use of multiple color models, the color classification system provides increased flexibility.

[0042] In view of the following description, other advantages and benefits of the color classification system will become apparent. In addition, as shown in the previous discussion, the present disclosure utilizes various terms to describe the features and advantages of the color classification system. Before describing the color classification system with reference to the following figures, additional details are now provided regarding the meaning of these terms.

[0043] As used herein, the term "digital image" (or simply "image") refers to a digital graphics file that, when drawn, displays one or more objects. In particular, an image can include one or more objects associated with any suitable object type or object class. In various embodiments, the image selection system displays the image on a computing device, such as a client device. In additional embodiments, the image selection system enables a user to modify or change an image, generate a new image, and search for images.

[0044] As used herein, the term "object" refers to a visual representation of a subject in an image. In particular, an object refers to a collection of pixels in an image that combine to form a visual depiction of an item, object, or element. Objects can correspond to a wide variety of categories and concepts. In some embodiments, an image includes multiple instances (e.g., occurrences) of an object. For example, an image of a bouquet of roses includes multiple instances of the roses.

[0045] In various embodiments, as used herein, the term "query object" corresponds to an object identified in a query (e.g., a request to search for an object or a request to select an object). In particular, the query object can refer to an object in the query that is requested to be detected and / or selected. In some instances, nouns in a query often refer to the query object, while adjectives refer to attributes of the query object (e.g., object attributes), such as object color. As used herein, the term "query color" refers to a color identified in the query that corresponds to the query object. In practice, the query color refers to the color of the query object (or at least a portion thereof) to be detected or selected in an image.

[0046] As used herein, the term "color" refers to the property of light reflected at a particular wavelength. In addition to hue, color can also include attributes such as brightness, contrast, saturation, hue, shade, color quality, hue, colorimetry, luminosity, chromaticity, background, iridescence, intensity, pleochroism, and / or colorimetric quality. In many embodiments, each pixel in an image is shown as a color. Additionally, the color of a pixel is represented by a color value associated with a color model. A color model is associated with different multidimensional color spaces (e.g., 3 or more dimensions) corresponding to the values and attributes defined by the corresponding color model.

[0047] An example of a color model is the red, green, and blue ("RGB") color model, which uses a set of components 256x256x256 in hexadecimal code to form a color value definition. Another color model is CIELAB (or simply "LAB"), which expresses three components for a color - L* for lightness from black (0) to white (100), a* from green (-) to red (+), and b* from blue (-) to yellow (+). In addition, other color models include the HSL (hue, saturation, lightness) model, the HSV (hue, saturation, value) model, and CMYK (cyan, magenta, yellow, hue).

[0048] The term "alternative color" (or "alternative query color") refers to a derivative or alternative version of a color. For example, an alternative color is a copy of a given color in which one or more color values or attributes have been changed. A color classification system can generate one or more alternative colors for a given color. For example, an alternative color can include a color whose brightness, saturation, or lightness (or another attribute) has been modified. As described below, a color classification system can utilize multiple color models to generate alternative colors. For example, a color classification system can map a color from a first color space to a second color space, modify the attributes of the color in the second color space, and map the modified color back to the first color space to generate an alternative color.

[0049] As used herein, the term "color similarity region" refers to a region of a multidimensional color space associated with a reference color (or simply "color") and one or more candidate colors. For example, a color similarity region may include a region defined by a color point of the reference color and one or more additional color points of candidate colors corresponding to the reference color. A color similarity region may capture the reference color and visually similar colors.

[0050] As used herein, the term "color match score" refers to the level of correspondence between a first color and at least a second color. In some embodiments, the color match score is the level or amount (e.g., number or percentage) of correspondence between a first color (e.g., a target color) and multiple versions of a second color (e.g., an "alternative query color"). As described below, the color match score can be based on the number or percentage of valid pixels within an object. In some embodiments, the color match score can be based on the pixel color match strength.

[0051] The term "color match threshold" refers to a condition or value at which a pixel is considered to match a color (e.g., a target color). For example, the color match threshold indicates when the color value for a pixel is sufficiently similar to a color in a multidimensional color space to be considered a match. The minimum color match threshold refers to the lowest color value a pixel can have to be considered a matching color.

[0052] As mentioned above, in various embodiments, the color classification system can employ machine learning and various neural networks. As used herein, the term "machine learning" refers to the process of constructing and implementing algorithms that can learn from data and make predictions about it. Generally, machine learning can operate by building a model based on example inputs to make data-driven predictions or decisions, such as image-exposure training pairs within a training dataset of images. Machine learning can include neural networks (e.g., object detection neural networks, object masking neural networks, and / or object classification neural networks), data-based models, or a combination of networks and models.

[0053] As used herein, the term "neural network" refers to a machine learning model that can be adjusted (e.g., trained) based on inputs to approximate unknown functions. In particular, the term neural network can include a model of interconnected neurons that communicate and learn to approximate complex functions and generate outputs based on multiple inputs provided to the model. For example, the term neural network includes an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to use supervised data to adjust the parameters of the neural network to model high-level abstractions in the data. Examples of neural networks include convolutional neural networks (CNNs), regional CNNs (R-CNNs), faster R-CNNs, mask R-CNNs, and single shot detection (SSD).

[0054] Reference is now made to the accompanying drawings, Figure 1 1 illustrates a schematic diagram of an environment 100 in which a color classification system 106 may operate according to one or more embodiments. Figure 1 As shown in FIG, environment 100 includes a client device 102 and a server device 110 connected via a network 108. Additional details regarding computing devices (e.g., client device 102 and server device 110) are provided below in conjunction with Figure 13 To provide. In addition, Figure 13 Additional details regarding networks, such as the illustrated network 108, are also provided.

[0055] although Figure 1 While a specific number, type, and arrangement of components within environment 100 are illustrated, various additional environment configurations are possible. For example, environment 100 may include any number of client devices. As another example, server device 110 may represent a group of connected server devices. As another example, client device 102 may communicate directly with server device 110, bypassing network 108 or utilizing separate and / or additional networks.

[0056] As shown, environment 100 includes client device 102. In various embodiments, client device 102 is associated with a user (e.g., a user client device), such as a user who requests automatic selection of an object in an image of a specific color (e.g., a selection request) and / or requests detection of images that include objects of a target color (e.g., a search request). Client device 102 may include image selection system 104 and color classification system 106. In various embodiments, image selection system 104 implements color classification system 106. In alternative embodiments, color classification system 106 is separate from image selection system 104. Although image selection system 104 and color classification system 106 are shown on client device 102, in some embodiments, image selection system 104 and color classification system 106 are located remotely from client device 102 (e.g., on server device 110), as further explained below.

[0057] Typically, the image selection system 104 facilitates the search, creation, modification, and / or deletion of digital images within an application. Furthermore, the image selection system 104 can facilitate the maintenance, search, identification, and / or sharing of digital images. In one or more embodiments, the image selection system 104 provides various tools related to image creation and editing (e.g., photo editing). For example, the image selection system 104 provides selection tools, color correction tools, and image manipulation tools. In some embodiments, the image selection system 104 provides tools related to object detection within an image and across a collection of images. In various embodiments, the image selection system 104 can operate in conjunction with one or more applications to generate or modify images. For example, in one or more embodiments, the image selection system 104 operates in conjunction with digital applications such as Adobe Photoshop, Adobe Elements, Adobe InDeSign, Adobe Acrobat, Adobe Illustrator, Adobe Stock, Adobe After Effects, Adobe Premiere Pro, Creative Cloud software, Behance, or other image editing and search applications.

[0058] In some embodiments, the image selection system 104 provides an intelligent image editing assistant that performs one or more automated image editing operations for the user. For example, given an image of a group of wolves, the user requests the image selection system to "remove the white wolf." As part of fulfilling the request, the image selection system 104 utilizes the color classification system 106 to automatically detect the white wolf. The image selection system 104 can then utilize additional system components (e.g., an object masking network and / or a hole filling neural network) to select, remove, or replace the selected white wolf. In some embodiments, the color classification system 106 automatically removes the object in response to the user request.

[0059] In various embodiments, the image selection system 104 provides an intelligent image search interface that identifies requested objects of a specified color. For example, a user or system (e.g., a computer image search system) provides a query to identify images that include a red cup. To fulfill the query, the image selection system 104 utilizes the color classification system 106 to automatically detect images that include a red cup within one or more image databases.

[0060] As mentioned above, the image selection system 104 includes a color classification system 106. As described in detail below, the color classification system 106 accurately classifies objects by color. In various embodiments, the color classification system 106 identifies the color of an object in an image. In one or more embodiments, the color classification system 106 matches one or more objects in an image to a query color based on a user request (e.g., a user-provided query string). In some embodiments, the color classification system 106 can be included as part of a dynamic system that determines, based on the query object, which object detection neural network to utilize and which additional neural networks and / or models (e.g., the color classification system 106) to utilize to select query objects having a particular attribute (e.g., the query color).

[0061] As shown, the environment 100 also includes a server device 110. The server device 110 includes a color classification server system 112. For example, in one or more embodiments, the color classification server system 112 represents and / or provides similar functionality as described herein in conjunction with the color classification system 106. In some embodiments, the color classification server system 112 supports the operation of the color classification system 106 on the client device 102.

[0062] Furthermore, in one or more embodiments, the server device 110 may include all or a portion of the color classification system 106. In particular, the color classification system 106 on the client device 102 may download an application (e.g., an image editing application and / or a search interface from the color classification server system 112) or a portion of a software application (e.g., an image search function within an image editing application) from the server device 110.

[0063] In some embodiments, the color classification server system 112 may include a web hosting application that allows the client device 102 to interact with content and services hosted on the server device 110. To illustrate, in one or more embodiments, the client device 102 accesses a web page hosted by the server device 110, which automatically detects the color of objects in an image based on user input from the client device 102. As another example, the client device 102 provides an image editing application that provides an image and a selection query to the color classification server system 112 on the server device 110, the selection query including a query object and a query color. The color classification server system 112 then detects the query object of the query color and provides an object mask of the detected query object back to the client device 102. The image editing application on the client device 102 then selects the detected query object using the object mask.

[0064] As mentioned above, Figure 1 An example environment 100 is shown in which the color classification system 106 may operate. Turning to the next figure, Figure 2 Provides an overview of color classification systems for automatically detecting colored objects in images. In particular, Figure 2 A series of actions 200 are illustrated for classifying a query object within a digital image as a target color (e.g., a query color) within the digital image, according to one or more embodiments. In various embodiments, the color classification system 106 performs the series of actions 200. In additional embodiments, additional systems, such as the image selection system 104, an object detection neural network, and / or an object masking system, perform the steps in the series of actions 200. Figure 2 As shown in FIG, a series of actions 200 includes an action 202 in which the color classification system 106 identifies a request for an object of a target color. Figure 2 As shown in conjunction with action 202, the user provides the query string "gray hat". For example, the user is requesting the color classification system 106 to identify images including gray hats via an image search system. In another example, the user is requesting the automatic selection of gray hats within an image being edited by the user via an image editing program.

[0065] In various embodiments, the color classification system 106 may provide a graphical user interface that enables a user to enter a search or select a query. For example, the graphical user interface enables a user to enter text input (e.g., a query string) indicating a query object and a query color. In some embodiments, the graphical user interface provides a color selection menu (e.g., a drop-down menu, a color wheel, a color gradient, a color palette, or other color selection interface) from which a user can select a color from a color set.

[0066] In response to receiving the query request, the color classification system 106 can perform an action 204 of mapping the query color to a plurality of points in a multidimensional color space. For example, the color classification system 106 converts the query color into color values defined by a color model and plots the query color in the multidimensional color space. Typically, the multidimensional color space includes at least three dimensions, but can include any number of dimensions, where each dimension can correspond to an attribute or characteristic of a given color.

[0067] Additionally, as part of performing action 204, the color classification system 106 may identify candidate query colors. For example, the color classification system 106 generates and / or identifies variants of the query color and maps those colors to a multidimensional color space. In this manner, the color classification system 106 generates a color similarity region for the query color in the multidimensional color space. As shown in conjunction with action 204, Figure 2 A simplified mapping of a query color (ie, QC) and candidate query colors (ie, AC1 and AC2) in a color space is illustrated.

[0068] As described below, in some embodiments, the color classification system 106 utilizes the second color model to generate one or more candidate query colors. For example, the color classification system 106 converts the query color into a second color value defined by the second color model, modifies one or more attributes of the query color based on the second color model, and then converts the modified query color back to the first color model to generate the candidate query colors. In addition, the color classification system 106 can map each candidate query color to a multidimensional color space of the first color model. Figure 4C Provides additional details about mapping a query color to multiple points in a multidimensional color space.

[0069] In an alternative embodiment, the color classification system 106 may access previously generated color similarity regions for the query color. For example, the color classification system 106 previously performed act 204 of mapping the query color to a plurality of points in the multidimensional color space. Furthermore, in some embodiments, the color classification system 106 has already generated color similarity regions for a plurality of colors (e.g., the most popular or frequently requested colors).

[0070] The series of acts 200 may include an act 206 in which the color classification system 106 detects a query object in the image. In various embodiments, the color classification system 106 utilizes one or more object detection neural networks or other models to detect the query object within the image. In one or more embodiments, one or more other systems perform act 206, such as an object selection system, an object detection system, and / or an object masking system as part of an image selection system or a remote or third-party system. In some embodiments, the color classification system 106 identifies one or more target regions in the image (e.g., pixels in the image associated with the detected object). Figures 9A-9D Provides additional details on detecting query objects from queries in images.

[0071] As shown in conjunction with action 206, Figure 2 An image of two boys wearing hats is shown, where two hats are detected. For example, color classification system 106 determines from the query that the query object is "hat." Based on the query object, color classification system 106 identifies two instances of hats in the image. In practice, when multiple instances of the query object are included in the image, color classification system 106 can detect each of the multiple instances.

[0072] As shown, the series of actions 200 may include an action 208 in which the color classification system 106 determines a color match between the detected query object and the query color. For example, in some embodiments, the color classification system 106 may compare the query color to the color (e.g., pixels) of the detected query object to determine whether the detected query object matches the query color. As described in detail below, the color classification system 106 may plot the color of the detected query object into a multidimensional color space to determine whether the color of the detected query object falls within a color similarity region associated with the query color.

[0073] As shown in conjunction with action 208, Figure 2 A simple depiction of the color space is shown. When the color classification system 106 applies a color similarity threshold to each mapped point (e.g., shown as a circle around each point), the query color and the candidate query colors form a color similarity region within the color space. As mentioned above, this color similarity region within the color space is better aligned with human perception of color than a simple single color similarity threshold around the query color alone. Figures 5A-5E Provides additional details about determining a color match between a detected query object and a query color.

[0074] In addition, Figure 2, action 208 shows where the gray hat and the white hat appear in the multidimensional color space. As shown, the gray hat is within the color similarity threshold of the query color and the candidate query color, indicating that the color matches the query color. In addition, the white hat is outside the color similarity threshold of the query color and the candidate query color, indicating that no color matches the query color.

[0075] In some embodiments, the color classification system 106 performs action 206 based on comparing pixels of the query object to a plurality of previously generated color similarity regions (or a plurality of color similarity regions generated along with generating color similarity regions for the query color). In some embodiments, the color classification system 106 can determine a color match between the detected query object and the query color based on pixels of the query object having the highest correspondence with the color similarity region of the query color compared to color similarity regions of other colors. For example, more pixels of the query object are located within the color similarity region of the query color than within color similarity regions corresponding to other colors. In another example, pixels of the query object are located closer to the query color and / or candidate query colors in the multidimensional color space than to the other colors and / or corresponding candidate colors.

[0076] As shown, a series of actions 200 may include an action 210 in which the color classification system 106 provides images of objects that match a query color. In some embodiments, the color classification system 106 may provide images, for example, in response to an image search query. In alternative embodiments, the color classification system 106 may provide images of query objects that match a selected query color to a user, such as within an image editing application. In one or more embodiments, the color classification system 106 makes a selection (e.g., provides an object mask of a gray hat).

[0077] By way of illustration, the color classification system 106 can perform some of the actions 202-210 in the series of actions 200 in various orders. For example, the color classification system 106 can detect query objects (e.g., action 206) before, during, or after mapping the query color and the candidate query colors to the multidimensional color space (e.g., action 204). Furthermore, the series of actions 200 can include additional actions, such as generating an object mask for each detected query object.

[0078] As mentioned above, the color classification system 106 can map the color to a multidimensional color space. Additionally, the color classification system 106 can generate candidate colors related to the color and map them to a multidimensional color space. Figures 3A-4C The mapping of colors and color candidates to a multi-dimensional color space and the corresponding actions performed by the color classification system 106 are illustrated. In particular, Figure 3A-Figure 3BThe color classification system 106 is illustrated mapping colors to a color space in an image in accordance with one or more embodiments. Figures 4A-4C The color classification system 106 is illustrated as generating candidate colors and mapping them to a color space in accordance with one or more embodiments.

[0079] For ease of explanation, Figure 3A-Figure 10 Embodiments of the present invention are described in terms of query colors and query objects. However, in many of the embodiments described below, the color classification system 106 may pre-generate, store, and utilize one or more color similarity regions corresponding to multiple colors (e.g., multiple query colors or a common set of colors) in addition to the query color. Furthermore, the color classification system 106 may generate multiple color similarity regions for multiple colors in conjunction with (e.g., simultaneously or nearly simultaneously with) generating the color similarity region for the query color.

[0080] Now turn Figure 3A-Figure 3B , Figure 3A A color mapping table 302 is illustrated. To provide context, in various embodiments, the color classification system 106 can convert a query color into a color value. In various embodiments, the color classification system 106 can utilize the color mapping table 302 to convert the query color into a color value. As shown, the color mapping table 302 includes a first column of known colors organized by color names 304 and a second column of color values 306 corresponding to the color names 304 (e.g., colors). Typically, the color mapping table 302 corresponds to a first color model (e.g., a LAB color model or another color model) that specifies the color components that make up the color values 306. In practice, the color values 306 can include mathematical values (e.g., LAB color values) for the components of each color included in the color model.

[0081] Once the query color is identified, the color classification system 106 can convert the query color into a color value within the color model. For example, if the color classification system 106 identifies the query color as "blue," the color classification system 106 can use the color mapping table 302 to find the color name 304 "blue" and the corresponding color value 306 having the components "w2, x2, y2, and z2." In various embodiments, the color value 306 includes components that are shown as a number, a percentage, a value range, and / or a set of axis coordinates.

[0082] Based on the color value 306, the color classification system 106 can map the query color to a multidimensional color space (or simply "color space"). For illustration, Figure 3BA simplified version of the color space (i.e., color space 310) is shown. Although a simplified two-dimensional color space is illustrated for ease of explanation, color space 310 may include additional dimensions. For example, color space 310 may include three dimensions that indicate three different properties and characteristics of color within a color model. In some embodiments, color space 310 is generated by a color embedding neural network (i.e., a machine learning model) and includes a large number of dimensions corresponding to the underlying characteristics of color.

[0083] like Figure 3B , the color classification system 106 can map the query color to the color space 310 (shown as mapped query color 312). In various embodiments, the color classification system 106 utilizes the color values 306 (i.e., components) from the color mapping table 302 to identify axis coordinates within the color space 310. In alternative embodiments, the color classification system 106 converts the color values 306 from the color mapping table 302 to the axis coordinates to generate a color map for the query color.

[0084] As also shown, the mapped query color 312 is surrounded by a color similarity threshold 314 (e.g., a distance threshold). The color similarity threshold 314 can indicate whether the other color is similar to or a match to the query color. For example, if the other color is mapped within the color similarity threshold 314 of the query color, the color classification system 106 can determine that the other color is a match.

[0085] Generally, color similarity threshold 314 is represented by a specified distance within color space 310. Color similarity threshold 314 can form various shapes based on the number of dimensions in color space 310. For example, if color space 310 is three-dimensional, color similarity threshold 314 will form a sphere. Similarly, if color space 310 is n-dimensional, color similarity threshold 314 will form an n-dimensional shape.

[0086] As mentioned above, once a query color is identified, color classification system 106 can also identify one or more alternative query colors that also map to color space 310. To this end, FIG4A illustrates a series of actions 400 for generating alternative query colors. In various embodiments, color classification system 106 can perform series of actions 400.

[0087] As shown, a series of actions 400 may include an action 402 in which the color classification system 106 obtains color values 306 (e.g., components) for a query color with respect to a first color model. In various embodiments, the color classification system 106 may utilize a color mapping table, as described above. For example, if the query color is identified from a query string, the color classification system 106 may utilize the color mapping table 302 to identify the color values 306 for the query color. In alternative embodiments, the color classification system 106 may identify the color values 306 for the query color from metadata associated with the query color. For example, if the query color is selected from a discrete set of colors, the color classification system 106 identifies the color values 306 that have been associated with the query color.

[0088] Additionally, the series of actions 400 may include an action 404 in which the color classification system 106 converts the color value 306 from the first color model to the second color model. In one or more embodiments, the color classification system 106 may utilize a color conversion function to convert the components of the color value 306 of the query color from the color model associated with the color space 310 to the second color model. For example, the color classification system 106 may utilize a LAB to HSL color conversion function to convert the query color from a LAB color value to an HSL color value.

[0089] In one or more embodiments, color classification system 106 can convert the copy of the query color to a second color model. In other words, even though color classification system 106 converts the query color to a second color model, color classification system 106 still retains a record of the color value of the query color. In this manner, the converted copy of the query color becomes an alternative query color that complements the original query color.

[0090] like Figure 4A As shown in FIG4 , a series of actions 400 may include an action 406 in which the color classification system 106 modifies the converted color value in the second color model to generate an alternative color value. In various embodiments, the color classification system 106 may modify one or more properties of the converted query color by changing a component of the color value using the second color model. For example, if a component of the second color model corresponds to color brightness, the color classification system 106 may decrease or increase the component value of the converted query color without changing other components, as shown in action 406.

[0091] In various embodiments, the color classification system 106 can perform one or more predetermined modifications. For example, continuing with the above example, the color classification system 106 can reduce or increase the brightness component of the converted query color by a certain percentage (e.g., half or 20%). In an alternative embodiment, the color classification system 106 reduces or increases the brightness component by a value (e.g., 10 points). In many embodiments, the color classification system 106 creates multiple copies of the converted query color and performs a separate modification on each copy (e.g., increasing the brightness by 50% on one copy and decreasing the brightness by 50% on a second copy) to create a set of alternative query values.

[0092] In some embodiments, the modification is based on color and / or color value. For example, the color classification system 106 may determine that a first color has a greater modification tolerance than a second color with respect to a component of a second color model. In various embodiments, the color classification system 106 enables a user to specify a color matching tolerance that affects the amount of modification (e.g., providing a scaling weight) that the color classification system 106 applies to the components of the converted query color in the second model. In one or more embodiments, the color classification system 106 utilizes machine learning based on training and feedback to learn the optimal amount to adjust the components in the second model.

[0093] In various embodiments, color classification system 106 can select which secondary color model to utilize based on which color attributes of the query color the color classification system 106 desires to modify. Indeed, some color models have specific components for a color attribute that other color models lack. Generally, a color model that focuses on a target color attribute can perform more granular modifications to that color attribute than other color models. For example, one color model can more precisely adjust a color's saturation, while another color model can better alter its brightness.

[0094] like Figure 4A , a series of actions 400 may include an action 408 in which the color classification system 106 converts the candidate color values from the second color model back to the first color model. In various embodiments, the color classification system 106 may utilize the same or another color conversion function to map the components of the candidate color values of the second color model to the first color model (e.g., a color model associated with the color space 310) to generate the candidate query colors. For example, the color classification system 106 may convert the candidate color values from the HSL color model back to the LAB color model. As mentioned above, in some embodiments, the color classification system 106 may convert the set of candidate color values created in the second model to generate a set of candidate query colors corresponding to the first color model and the first color space.

[0095] As mentioned above, the color classification system 106 can use one secondary color model to adjust one color attribute of the query color and use another secondary color model to adjust another color attribute of the query color. In some embodiments, the color classification system 106 uses multiple secondary color models to modify multiple color attributes of the query color. For example, the color classification system 106 converts the query color to a second color model and modifies the first color attribute, and then converts it back to the first color model to generate an alternative query color. The color classification system 106 then converts the alternative query color (or a copy) to a third color model, modifies the second color attribute, and converts it back to the first color model (e.g., replacing or adding it to the alternative query color). In addition, in some embodiments, the color classification system 106 uses a color model associated with the color space 310 to modify one or more color attributes of the query color. For example, the first color model can include RGB, the second color model can include HSL, and the third color model can include LAB. In this way, the color classification system 106 can modify the saturation of a color in the HSL color model to generate a first alternative color and map the first alternative color back to the RGB space. Similarly, the color classification system 106 may modify the lightness of the color in the LAB space to generate a second candidate color and map the first candidate color back to the RGB space.

[0096] In various embodiments, the color classification system 106 may update the color mapping table 302 to include an alternative query color to the query color. Figure 4B An updated color mapping table 302' is shown with additional entries added to the columns corresponding to color names 304 and color values 306. In particular, the updated color mapping table 302' shows additional entries for alternative query colors and corresponding color values associated with the query color "blue." In some embodiments, the names of the alternative query colors may indicate modified properties of the alternative query colors (e.g., "Blue_Alt1_Brightness-50%").

[0097] By maintaining an updated color mapping table 302′, the color classification system 106 can reduce the need to generate alternative query colors when the same query color is identified in future queries. In fact, instead of regenerating one or more alternative query colors, the color classification system 106 can utilize the updated color mapping table 302′ to determine that an alternative query color is already associated with the query color. In an alternative embodiment, the color classification system 106 generates a new alternative query color each time a query color is identified.

[0098] As mentioned above, once candidate query colors for a query color are generated, the color classification system 106 can add the candidate query colors to the color space 310. For illustration, Figure 4C The color classification system 106 is shown mapping the candidate query colors to the color space 310. As shown, the mapped candidate query colors 412 (i.e., AC1, AC2, AC3, and AC4) are mapped near the mapped query color 312 (i.e., QC) because the candidate query colors represent variations of the query color. In addition, the color space 310 shows a color similarity threshold 414 (e.g., a distance threshold) associated with the mapped candidate query colors 412.

[0099] In one or more embodiments, the size (e.g., distance) of the color similarity threshold 414 for the mapped candidate query color 412 is different from the color similarity threshold 314 for the mapped query color 312. For example, the size of the color similarity threshold 414 for the mapped candidate query color 412 is smaller. In some embodiments, the size of the color similarity threshold 414 for the mapped candidate query color 412 is based on the amount of modification performed on the second color model by the color classification system 106. For example, the color similarity threshold for a first candidate query color whose components are modified by 50% is smaller than the color similarity threshold for a second candidate query color whose components are modified by 20% (e.g., a larger modification may correspond to a larger variance from the query color, thereby leading to a false positive result).

[0100] In various embodiments, the mapped query color 312, the mapped candidate query color 412, and their corresponding color similarity thresholds 314, 414 form a color similarity region 416 that defines the color space 310, indicating color similarity to the query color. In practice, if a color is mapped to a point within the color similarity region 416, then the color classification system 106 can determine that the color matches the query color.

[0101] Although the color similarity region 416 is shown as a simple two-dimensional outline, the color similarity region 416 may include holes or patches within the color similarity region 416 where the color similarity thresholds 314, 414 do not intersect. Additionally, the color similarity region 416 may be shaped to match the number of dimensions in the color space 310. For example, in a three-dimensional color space, the color similarity region 416 may include a union of multi-dimensional spheres that intersect one another.

[0102] Figures 4A-4C Various embodiments are described for generating query colors and corresponding candidate query colors and mapping them to a multidimensional color space. Figures 4A-4CThe described actions and algorithms provide example structures and architectures for performing the steps of generating color similarity regions for a target color in a multidimensional color space. For example, in combination with at least Figure 4A The described flowchart provides structure and / or action for one or more algorithms corresponding to the color classification system 106 generating a color similarity region for a first color in a multi-dimensional color space.

[0103] As mentioned above, once the color classification system 106 can generate the color similarity region 416 for the query color in the color space 310 and identify the detected query object, the color classification system 106 can determine whether the detected query object matches the query color. Because the detected query object is composed of a certain number of pixels, each of which can correspond to a different color, the color classification system 106 can determine a color match score for the detected query object based on multiple (some or all) pixels of the detected query object.

[0104] to this end, Figures 5A-5E 1 shows that the color classification system 106 generates a color matching score for a query object according to one or more embodiments. Figure 5A Detected query objects 502 (e.g., flowers) or instances of detected query objects are illustrated. For example, the color classification system 106 detects a query for "blue flowers" and, in response, detects the query object 502 within the image. In various embodiments, the color classification system 106 also generates an object mask for the detected query object 502 to further isolate the pixels of the detected query object 502 from the background pixels of the image, which are typically of a different color. Figures 9A-9D Further description is given for detecting query objects and generating object masks.

[0105] As mentioned above, the detected query object 502 is composed of individual pixels. For ease of explanation, Figure 5A The diagram shows an enlarged set of pixels 504 representing a portion of pixels constituting a detected query object 502. The enlarged set of pixels 504 is organized into rows and columns. In addition, two representative pixels, a first pixel 506 and a second pixel 508, are called out in the enlarged set of pixels 504.

[0106] In one or more embodiments, the color classification system 106 may generate a color matching score based on a percentage of valid pixels within the detected query object 502. To provide context, in many embodiments, the valid pixels are within an object mask (i.e., a binary mask) corresponding to the detected query object 502 and are within the color similarity region 416 in the color space 310 (e.g., match the query).

[0107] To illustrate, Figure 5B The color space 310 described above is shown, including a mapped query color 312 , a mapped candidate query color 412 , and a color similarity region 416 . Figure 5B Also included is a mapping of the second pixel 508 and the first pixel 506 within the color space 310 of the first color model. As shown, the first pixel 506 is mapped outside the color similarity region 416, while the second pixel 508 is mapped inside the color similarity region 416. Therefore, the color classification system 106 can determine that the first pixel 506 is invalid (although within the object mask) and the second pixel 508 is valid.

[0108] In one or more embodiments, the color classification system 106 can map each pixel of the detected query object 502 to the color space 310 to determine whether the pixel is valid and matches the query color. In some embodiments, the color classification system 106 maps a subset of pixels. For example, the color classification system 106 systematically maps every other pixel, every third pixel, or every two of every four pixels. In another example, the color classification system 106 selects a random selection of pixels until a predetermined number of pixels (e.g., 40% or 250 pixels of the detected query object) are reached.

[0109] In various embodiments, the color classification system 106 can reduce or downsize the detected query object 502 before mapping the pixels. For example, the color classification system 106 can combine adjacent pixels or merge the colors of adjacent pixels by selecting a representative pixel. For example, the color classification system 106 selects pixel A1 in the enlarged pixel set 504 to represent the entire set. In this manner, the color classification system 106 can reduce the number of pixels of the detected query object 502, which in turn reduces the processing and memory requirements required to determine the color match score. This improvement in computational efficiency can be significant when processing large numbers of images and / or large image formats.

[0110] For each pixel detected in the query object 502, the color classification system 106 can record whether the pixel is valid or invalid. To illustrate, Figure 5C Pixel validity table 510 corresponding to the enlarged pixel set 504 is shown. Figure 5C , the pixel validity table 510 includes the following columns: pixel 512 (e.g., pixel identifier), valid 514, and invalid 516. In practice, based on mapping the pixel to the color space 310 and determining whether the pixel falls within the color similarity region 416, the color classification system 106 can quickly identify each pixel in the detected query object 502 as valid or invalid.

[0111] In some embodiments, the color classification system 106 does not maintain the color similarity region 416 as shown. Instead, for a given pixel of a detected query object 502, the color similarity region 416 determines the distance between the given pixel and the locations of the mapped query color 312 and the mapped candidate query color 412. If one of the locations is within one of the color similarity thresholds 314, 414, the color classification system 106 determines the pixel as a valid color match. Otherwise, the color classification system 106 determines the pixel as invalid.

[0112] As mentioned above, in one or more embodiments, the color classification system 106 can determine a color match score for the detected query object 502 based on the amount (e.g., number or percentage) of pixels determined to be valid. In some embodiments, the color classification system 106 can determine the color match score as a percentage of valid pixels. For illustration, the pixel validity table 510 indicates a percentage of total valid pixels (e.g., 75%) as the color match score 518. In alternative embodiments, the color classification system 106 can generate a numerical value for the color match score based on the number of valid pixels. For example, the color classification system 106 can add 1 point to the numerical value for every 1 or 5 valid pixels identified. In some cases, the color classification system 106 normalizes the numerical value (e.g., between 1 and 100).

[0113] As described above, in various embodiments, the color classification system 106 may determine a color match score based on a percentage of pixels determined to be valid in the detected query object 502. In alternative embodiments, the color classification system 106 may use other and / or additional metrics to determine a color match score. For illustration, Figure 5D Pixel color matching table 520 corresponding to the enlarged pixel set 504 is shown. Figure 5D As shown in , pixel color matching table 520 includes the following columns: pixel 522 (eg, pixel identifier), shortest distance 524 , average distance 526 , and query color distance 528 .

[0114] In the above embodiment, shortest distance 524 may correspond to the shortest distance in color space 310 between a given pixel of detected query object 502 and mapped query color 312 or the closest mapped alternative query color 412. Average distance 526 may correspond to the average distance in color space 310 between a given pixel and mapped query color pixels (e.g., mapped query color 312 and mapped alternative query color 412). Query color distance 528 may correspond to the distance in color space 310 between a given pixel and mapped query color 312.

[0115] In various embodiments, the color classification system 106 may generate a color match score based on one or more metrics corresponding to the shortest distance 524, the average distance 526, and / or the query color distance 528. For example, the color classification system 106 compares the query color distance 528 to a query color distance threshold to determine the shortest distance 524. In alternative embodiments, the color classification system 106 may generate a color match score based on a combination of the shortest distance 524, the average distance 526, and / or the query color distance 528. For example, the color classification system 106 compares each metric to a corresponding distance threshold and then combines the calculations to generate a color match score. In additional embodiments, the color classification system 106 may further weight each metric when calculating the color match score. For example, the color classification system 106 may weight the query color distance 528 more heavily (e.g., making it more influential) than other metrics.

[0116] Based on the color match score, the color classification system 106 can determine whether the color of the detected query object 502 (or detected instance of the query object) matches the query color as a whole. As mentioned above, in some embodiments, if the color match score has a percentage (e.g., pixel percentage or normalized value) or number of pixels above a color match threshold, then the overall color of the detected query object matches the query color. In one example, if the color match score is above a color match percentage threshold of 50% (e.g., a majority of pixels match), then the color of the detected query object 502 is determined to match the query color. In other examples, if the color match score is above a color match value threshold (e.g., 50%, 75%, 80%, or 90%), then a color match exists.

[0117] In various embodiments, the user may specify a tolerance value for the match. For example, based on the user lowering the color match tolerance, the color classification system 106 may lower the color match percentage threshold (e.g., requiring 60% of the pixels to match). Similarly, if the user tightens or increases the color match tolerance, the color classification system 106 may increase the color match percentage threshold (e.g., requiring 80% of the pixels to match).

[0118] Figure 5E A flow chart is shown corresponding to a series of actions 500 for generating a color matching score for a query object, as described above with respect to Figures 5A-5D In particular, in combination Figure 5EThe described actions and algorithms provide example structures and architectures for one or more algorithms that correspond to performing steps for determining that a detected object matches a target color (i.e., a first color). For example, the series of actions 500 may include an action 530 in which the color classification system 106 determines the validity of pixels in the detected query object 502 (as described above in conjunction with Figures 5A-5C Additionally, the series of actions 500 may include an action 532 in which the color classification system 106 analyzes the percentage of valid pixels in the detected query object 502 (described above in conjunction with Figure 5C-5D described).

[0119] In one or more embodiments, based on determining that the color matches the query color, color classification system 106 can provide detected query objects 502 or detected instances of the query object in response to the query. In embodiments where multiple instances of the query object are detected, color classification system 106 can apply additional filters to enhance the quality of the query results.

[0120] To illustrate, Figure 6 A flowchart of filtering query object instances based on color matching scores according to one or more embodiments is illustrated. In particular, Figure 6 Illustrated is a series of actions 600 for selecting the best color matching query object instance to provide to the user. In various embodiments, the color classification system 106 can perform the series of actions 600.

[0121] As shown, a series of acts 600 may include an act 602 in which the color classification system 106 determines multiple instances of a query object are detected. In some embodiments, multiple query object instances are detected within the same image. In one or more embodiments, multiple query object instances are detected across multiple images. For example, as shown in act 602, the color classification system 106 detects a query for "yellow tree," where the query object is "tree" and the query color is "yellow," and two instances of the tree are detected in each image.

[0122] As shown, a series of actions 600 may include an action 604 in which the color classification system 106 generates a color matching score for each query object instance. Figure 5A - Figure 5D As described above, color classification system 106 can determine a color matching score for each of the plurality of query object instances. For example, as described above, color classification system 106 can determine the color matching score based on a percentage of pixels (e.g., valid pixels) that match the query color determined for each query object instance.

[0123] As shown, sequence of actions 600 may include action 606 in which color classification system 106 filters out query object instances based on a minimum color match threshold. As described above, color classification system 106 may compare the color match score of each query object instance to the minimum color match threshold to determine which query object instances do not match the query color. For example, as shown in action 606, by applying a minimum color match threshold of 50%, color classification system 106 may filter out the lowest query object instances.

[0124] Furthermore, as shown, sequence of actions 600 may include action 608 in which color classification system 106 filters out query object instances that exceed a color deviation threshold. For example, in one or more embodiments, color classification system 106 identifies the query object instance with the highest color match score (i.e., 96 in action 608). Additionally, color classification system 106 determines whether any query object instance has a color match score below the color deviation threshold. In this manner, in some embodiments, the color deviation threshold serves as an additional color match threshold.

[0125] For illustration, action 608 shows a first query object instance with a color match score of 52, a second query object instance with a color match score of 70, a third query object instance with a color match score of 96, and a color deviation threshold of 30. Here, color classification system 106 identifies the third query object instance as having the highest color match score (96). Next, color classification system 106 applies the color deviation threshold to the highest color match score to establish a new minimum color match threshold of 66. As a result, color classification system 106 filters out the first query object instance (53), which is less than the new minimum color match threshold, but retains the second query object instance (70) and the third query object instance (96).

[0126] By applying a color deviation threshold, the color classification system 106 can improve the quality of the results returned in response to a query. For example, the color deviation threshold ensures that when multiple query object instances are detected, the best match and close second choices are provided as results. In alternative embodiments, the color classification system 106 can return the top result, the top x results, or the top y percentage of results.

[0127] As shown, the series of actions 600 may include an action 610 in which the color classification system 106 provides the remaining query object instances. For example, the color classification system 106 may provide the remaining query object instances to the user via the client device in response to the user providing the query. In another embodiment, the color classification system 106 may provide the remaining query object instances as results to a system (e.g., an image search system) in response to a color matching object search request. As shown in action 610, the color classification system 106 provides images of the two query object instances with the two highest color matching scores.

[0128] Now turn Figure 7A and Figure 8B , provides additional details regarding using the color classification system 106 to detect color matching query objects in an image. Figures 7A-7C relates to search queries in image search systems. In particular, Figures 7A-7C It involves search queries in image search systems. As described below, Figure 8A-8B Involves a select query within an image editing application.

[0129] As shown in the figure, Figures 7A-7C A client device 700 having a graphical user interface 702 is illustrated. Figures 7A-7C The client device 700 in the embodiment may represent the above Figure 1 The client device 102 is introduced. As also shown, the graphical user interface 702 includes an image search interface 704, which includes a query input field 706 in which the user provides a color matching object query. In addition, the image search interface 704 may include an option to submit the query (e.g., a "Search" button). In some embodiments, the image search interface 704 may also include a color selection element in which the user can select a color from a color set.

[0130] like Figure 7A As shown in , graphical user interface 702 includes image collection 708 associated with image search interface 704. For example, the images are part of an image library associated with the user. In an alternative embodiment, graphical user interface 702 does not display the images until image search results are provided to the user. Figure 7A As shown in , the color classification system 106 detects a query for "evergreen tree." For example, as described above, the color classification system 106 can parse the query string to determine that the query object is "tree" and the query color is "evergreen."

[0131] In response to the query, the color classification system 106 may identify images in the image collection that include the query object. For illustration purposes, Figure 7BMultiple query object instances of trees within the image collection 708 are shown, represented by bounding boxes 710 surrounding the detected trees. However, in many embodiments, the color classification system 106 does not display the query object instances to the user. Figure 7B Instead, the color classification system 106 jumps to Figure 7C Graphical user interface 702 shown in .

[0132] like Figure 7C As shown in FIG, the graphical user interface 702 may include an image search interface 704 and a subset 712 of the image collection. In particular, the subset 712 of images includes search result images having one or more trees that match the query color (i.e., "evergreen") as determined by the color classification system 106. In practice, as described above, the color classification system 106 generates a query object instance (defined by Figure 7B In addition, as described above, the color classification system 106 can determine which images have query object instances that match the query color based on their corresponding color matching scores.

[0133] As mentioned above, Figure 8A-8B Involves a select query within an image editing application. Specifically, Figure 8A-8B A graphical user interface for detecting a query object in a digital image that matches a query color is illustrated in accordance with one or more embodiments. Figure 8A-8B A client device 800 includes a graphical user interface 802 displaying an image editing application. In various embodiments, Figure 8A-8B The client device 800 in the embodiment may represent the above Figure 1 The client device 102 is introduced. For example, the client device 800 includes an image editing application that implements the image selection system 104 and the color classification system 106. For example, the image editing application may generate Figure 3A-Figure 3B The graphical user interface 802 in.

[0134] like Figure 8A As shown in FIG, graphical user interface 802 includes an image 804 within an image editing application. For example, image 804 shows three pairs of shoes. For ease of explanation, image 804 is simplified to not include a background or other objects.

[0135] In addition, graphical user interface 802 includes an object selection interface 806 in which a user can request the image editing application to automatically detect and select objects within the image, including objects of a target color. As shown, object selection interface 806 includes a query field in which the user can enter a query string (i.e., "red shoes") and an option to request selection of the query (i.e., an "OK" element) or cancel object selection interface 806 (i.e., a "Cancel" element). In some embodiments, object selection interface 806 includes additional elements, such as a selectable option to select a color from a set of colors.

[0136] Based on receiving a selection query including a query string (i.e., "red shoes"), the color classification system 106 can automatically detect and select the query object. In particular, the color classification system 106 can detect each instance of the query object 808 (e.g., shoes) in the image 804 and identify the specific instance specified in the query (e.g., "red" shoes). To illustrate, Figure 8B The result of color classification system 106 automatically selecting red shoes 810 within image 804 (and / or selecting a target region of pixels / object mask corresponding to red shoes 810) in response to a selection request is shown. Once selected, the image editing application can enable the user to edit, copy, cut, move, and / or otherwise modify the selected object.

[0137] As mentioned above, in various embodiments, color classification system 106 can detect a query object (e.g., an instance of a query object) within an image and / or multiple instances of the query object detected. In some embodiments, color classification system 106 can utilize an object detection neural network to detect the query object instances and / or can utilize an object mask neural network to select the detected query object, as described below in conjunction with the following figures.

[0138] To illustrate, Figures 9A-9C A graphical user interface 902 of an image editing application is shown that enables a user to request selection of objects in an image, including objects of a target color. For ease of explanation, Figures 9A-9C The client device 800 introduced above is included. For example, the client device 800 includes an image editing application that implements the image selection system 104 and the color classification system 106.

[0139] like Figure 9A As shown in FIG, graphical user interface 902 includes an image 904 within an image editing application. Image 904 shows three cars 907, with the car on the right being white. As also shown, the image editing application includes various tools (e.g., a vertical toolbar) with selection options and other image editing options. In addition, graphical user interface 902 includes object selection interface 906, as described above in conjunction with Figure 8A As described, where the query is "white car".

[0140] As previously explained, once the user provides a query, the color classification system 106 can determine that the query object is "car" and the query color is "white" (e.g., using natural language processing to identify nouns and adjectives). Additionally, the object detection neural network can determine and automatically detect one or more instances of the query object in response to selecting the query using the object detection neural network.

[0141] To illustrate, Figure 9B Color classification system 106 is shown utilizing an object detection neural network to identify one or more instances of a query object within image 904. For example, if "car" is a known object, color classification system 106 can utilize a known object classification detection neural network to detect instances of cars (i.e., the query object) within image 904. In some embodiments, color classification system 106 utilizes a generalized object detection neural network to detect one or more instances of the query object in the image. In alternative embodiments, color classification system 106 determines to utilize a more specific, specialized object detection neural network that is specifically trained to detect a particular object type or object category.

[0142] Some examples of object detection neural networks include specialized object detection neural networks (e.g., sky detection neural networks, face detection neural networks, body detection neural networks, skin detection neural networks, and waterfall detection neural networks), object-based concept detection neural networks, known object category detection neural networks, and unknown object category detection neural networks. Examples of object detection neural networks may also include subnetworks and / or supporting object detection networks, such as object proposal neural networks, region proposal neural networks, and concept embedding neural networks.

[0143] like Figure 9B As shown in , color classification system 106 can generate an approximate boundary (e.g., bounding box 908) around detected instances of a query object. For example, as part of detecting one or more instances of a query object, an object detection neural network can create a boundary (e.g., a bounding box) around each query object instance. In some cases, the boundary provides a group or subset of pixels within image 904 that includes the corresponding query object instance.

[0144] like Figure 9CAs shown in , color classification system 106 can generate or otherwise obtain an object mask 910 for each detected query object instance. For example, color classification system 106 provides the detected objects to an object mask neural network, which generates an object mask (e.g., a selection mask) for the objects. In particular, color classification system 106 provides the bounding boxes of one or more query object instances to the object mask neural network. In some embodiments, color classification system 106 can downsample the pixels in the bounding boxes as described above.

[0145] When generating an object mask for a detected query object (or each detected query object instance), the object mask neural network can segment the pixels in the detected query object from the other pixels in the image. For example, the object mask neural network can create a separate image layer that sets the pixels corresponding to the detected query object to positive (e.g., binary 1) while setting the remaining pixels in the image to neutral or negative (e.g., binary 0). When this object mask layer is combined with image 904, only the pixels of the detected query object are visible. In effect, the generated object mask can provide a segmentation that supports selection of the detected query object within image 904.

[0146] The object mask neural network may correspond to one or more deep neural networks or models that select objects based on bounding box parameters corresponding to the objects within the image. For example, in one or more embodiments, the object mask neural network utilizes techniques and methods found in Ning Xu et al., “Deep GrabCut for Object Selection,” published July 14, 2017, the entire contents of which are incorporated herein by reference. For example, the object mask neural network may utilize a deep gradient approach rather than a saliency mask transfer. As another example, the object mask neural network can utilize techniques and methods found in: U.S. Patent Application Publication No. 2019 / 0130229, “DeepSalient Content Neural Networks for Efficient Digital Object Segmentation,” filed on October 31, 2017; U.S. Patent Application No. 16 / 035,410, “Automatic Trimap Generation and Image Segmentation,” filed on July 13, 2018; and U.S. Patent No. 10,192,129, “Utilizing Interactive Deep Learning To Select Objects In Digital Visual Media,” filed on November 18, 2015, each of which is incorporated herein by reference in its entirety.

[0147] like Figure 9D As shown in , the color classification system 106 can detect which detected query object instance matches the query color and provide the selection to the user. As described above, the color classification system 106 can generate a color matching score using each object mask based on comparing pixels within the object mask to the query color and the candidate query colors in the multidimensional color space.

[0148] In response to generating the color matching scores, the color classification system 106 may automatically select one or more query object instances that satisfy the selection request. Figure 9D As shown in , the color classification system 106 determines that the car on the right is white. Therefore, the color classification system 106 selects the white car 912 using the corresponding object mask and deselects or does not select the non-white car.

[0149] Now refer to Figure 10 , provides additional details regarding the capabilities and components of the color classification system 106 according to one or more embodiments. In particular, Figure 10A schematic diagram illustrating an example architecture of a color classification system 106 implemented within an image selection system 104 and hosted on a computing device 1000 is shown. The image selection system 104 may correspond to the color classification system 106 previously described in conjunction with Figure 1 The image selection system 104 is described.

[0150] As shown, the color classification system 106 is located on a computing device 1000 within the image selection system 104. In general, the computing device 1000 can represent various types of client devices. For example, in some embodiments, the client is a mobile device, such as a laptop, tablet, mobile phone, smartphone, etc. In other embodiments, the computing device 1000 is a non-mobile device or other type of client device, such as a desktop computer or server. Additional details about the computing device 1000 are provided below and in connection with the Figure 13 Have a discussion.

[0151] like Figure 10 As shown in FIG, color classification system 106 includes various components for performing the processes and features described herein. For example, color classification system 106 includes digital image manager 1010, user input detector 1012, object detection neural network manager 1014, object mask generator 1016, color matching manager 1018, and storage manager 1020. As shown, storage manager 1020 includes digital image 1022, object detection neural network 1024, object mask neural network 1026, multi-dimensional color space 1028, and color similarity region 1030. Each of the above-mentioned components is described below in turn.

[0152] As mentioned above, the color classification system 106 includes a digital image manager 1010. Generally, the digital image manager 1010 facilitates identifying, accessing, receiving, acquiring, generating, importing, exporting, copying, modifying, removing, and organizing images. In one or more embodiments, the digital image manager 1010 operates in conjunction with the image selection system 104 (e.g., an image search system and / or an image editing application) to access, edit, and search images, as previously described. In some embodiments, the digital image manager 1010 communicates with a storage manager 1020 to store and retrieve digital images 1022, for example, within a digital image database managed by the storage manager 1020.

[0153] As shown, the color classification system 106 includes a user input detector 1012. In various embodiments, the user input detector 1012 can detect, receive, and / or facilitate user input on the computing device 1000 in any suitable manner. In some instances, the user input detector 1012 detects one or more user interactions (e.g., a single interaction or a combination of interactions) with respect to a user interface. For example, the user input detector 1012 detects user interactions from a keyboard, a mouse, a touch screen, a touch screen, and / or any other input device associated with the computing device 1000. For example, the user input detector 1012 detects user input of a query (e.g., a selection query or an image search query) submitted from an object selection request interface requesting automatic detection and / or selection of color-matched objects within an image.

[0154] As shown, the color classification system 106 includes an object detection neural network manager 1014. In various embodiments, the object detection neural network manager 1014 maintains, creates, generates, trains, updates, accesses, and / or utilizes the object detection neural network disclosed herein. As described above, the object detection neural network manager 1014 detects one or more objects (e.g., query objects) within an image and generates boundaries (e.g., bounding boxes) to indicate the detected objects.

[0155] Additionally, in various embodiments, the object detection neural network manager 1014 can communicate with the storage manager 1020 to store, access, and utilize the object detection neural network 1024. As mentioned above, in various embodiments, the object detection neural network 1024 can include one or more specialized object detection neural networks, object-based concept detection neural networks, known object category detection neural networks, unknown object category detection neural networks, object proposal neural networks, region proposal neural networks, and concept embedding neural networks.

[0156] Additionally, as shown, the color classification system 106 includes an object mask generator 1016. In one or more embodiments, the object mask generator 1016 generates, creates, and / or produces accurate object masks from detected objects. For example, the object detection neural network manager 1014 provides the boundaries of an object (e.g., a detected query object) to the object mask generator 1016, which generates an object mask for the detected object using the object mask neural network 1026, as described above. Also as explained above, in various embodiments, when multiple instances of the query object are detected, the object mask generator 1016 generates multiple object masks.

[0157] As also shown, the color classification system 106 includes a color matching manager 1018. In some embodiments, based on a query color, the color matching manager 1018 determines, analyzes, detects, identifies, matches, maps, plots, filters, and / or selects one or more specific instances of a detected object from a plurality of detected instances of the object. In various embodiments, as described above, the color matching manager 1018 utilizes a multidimensional color space 1028 and / or one or more color models to identify instances of a target color.

[0158] In one or more embodiments, the color matching manager 1018 generates color similarity regions 1030 for a plurality of colors. For example, the color matching manager 1018 constructs one or more color similarity regions for each color in a set of colors (e.g., colors in a color palette). In this manner, when classifying the color of an object, the color matching manager 1018 can compare a pixel map from the object to the previously generated color similarity regions 1030 to determine a color match.

[0159] Each of the components 1010-1030 of the color classification system 106 can include software, hardware, or both. For example, the components 1010-1030 can include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as a client device (e.g., a mobile client device) or a server device. When executed by one or more processors, the computer-executable instructions of the color classification system 106 can cause the computing device to perform the feature learning method described herein. Alternatively, the components 1010-1030 can include hardware, such as a dedicated processing device that performs a specific function or group of functions. In addition, the components 1010-1030 of the color classification system 106 can include a combination of computer-executable instructions and hardware.

[0160] Furthermore, components 1010-1030 of the color classification system 106 can be implemented as one or more operating systems, one or more standalone applications, one or more modules of an application, one or more plug-ins, one or more library functions or functions that can be called by other applications, and / or a cloud computing model. Thus, components 1010-1030 can be implemented as standalone applications, such as desktop or mobile applications. Additionally, components 1010-1030 can be implemented as one or more web-based applications hosted on a remote server. Components 1010-1030 can also be implemented in a suite of mobile device applications or "apps." For illustration, components 1010-1030 can be implemented in applications including, but not limited to, Adobe Photoshop, Adobe Elements, Adobe InDesign, Adobe Acrobat, Adobe Illustrator, Adobe Stock, Adobe After Effects, Adobe Premiere Pro, Creative Cloud Software, and Behance. The foregoing are registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries.

[0161] Figures 1-10 , corresponding text and examples provide many different methods, systems, devices and non-transitory computer readable media of the color classification system 106. In addition to the foregoing, one or more embodiments may also be described in terms of a flowchart including actions for achieving a particular result, such as a flowchart. Figure 11 and Figure 12 Additionally, the actions described herein may be repeated or performed in parallel with each other or with different instances of the same or similar actions.

[0162] As mentioned, Figure 11 and Figure 12 11 and 1200 illustrate a flow chart of a series of actions 1100, 1200 for utilizing the color classification system 106 according to one or more embodiments. Figure 12 The actions according to one embodiment are illustrated, but alternative embodiments may omit, add, reorder, and / or modify Figure 11 and Figure 12 Any action shown in . Figure 11 and Figure 12 Alternatively, the non-transitory computer readable medium may include instructions that, when executed by one or more processors, cause the computing device to perform Figure 11 and Figure 12 In some embodiments, the system may perform Figure 11 and Figure 12 action.

[0163] Now turn Figure 11 In one or more embodiments, a series of actions 1100 is implemented on one or more computing devices, such as client devices 102, 700, 800, server device 110, or computing device 1000. Additionally, in some embodiments, the series of actions 1100 is implemented in a digital environment for classifying the color of an object in a digital image. In various embodiments, the series of actions 1100 is implemented in a digital environment for detecting instances of an object in a digital image. For example, the series of actions 1100 is implemented on a computing device having a memory that includes a digital image, a color similarity region for a color in a multidimensional color space that includes a plurality of mapped candidate color points mapped in the multidimensional color space, and an object mask neural network.

[0164] Series of acts 1100 may include an act 1110 of identifying color similarity regions for colors. In some embodiments, act 1110 may involve identifying one or more color similarity regions for one or more colors within a multidimensional color space. In various embodiments, act 1110 may include detecting an object using a trained object detection neural network. In one or more embodiments, act 1110 involves identifying a first color similarity region corresponding to a first color of the one or more colors, the first color similarity region comprising one or more mapped alternative versions of the first color mapped within the multidimensional color space.

[0165] In some embodiments, action 1110 may include generating an alternative version of the first color by converting a copy of the first color from a first color model corresponding to a multidimensional color space to a second color model, modifying one or more color properties of the copy of the first color within the second color model, and converting the copy of the first color with the modified one or more color properties from the second color model back to the first color model corresponding to the multidimensional color space.

[0166] In one or more embodiments, act 1110 may include generating a first color similarity region for a first color of the one or more colors, the region comprising a plurality of mapped candidate color points corresponding to the first color mapped to a multidimensional color space. In a particular embodiment, act 1110 may include identifying a query color and a corresponding query object from a query, wherein the query color corresponds to the first color, and wherein the query object corresponds to an object, and mapping the query color to a plurality of points in the multidimensional color space to generate the first color similarity region for the query color.

[0167] As shown, the series of actions 1100 also includes an action 1120 of identifying an object in the digital image. In particular, action 1120 may involve identifying an object in the digital image, the object comprising a plurality of pixels. In some embodiments, action 1120 may include utilizing an object mask neural network to isolate and identify the plurality of pixels associated with the object within the digital image.

[0168] like Figure 11 As shown in , a series of actions 1100 also includes an action 1130 of mapping pixels of the object to a multidimensional color space to determine color correspondences with color similarity regions. In particular, action 1130 may include mapping pixels of the plurality of pixels to the multidimensional color space to determine one or more color correspondences with one or more color similarity regions. In various embodiments, action 1130 may include mapping pixels of the plurality of pixels to within color similarity regions for colors to determine which colors the mapped object pixels correspond to.

[0169] As shown, the series of actions 1100 also includes an action 1140 of generating a color matching score for the object. In particular, action 1140 can include generating one or more color matching scores for the object based on one or more color correspondences between pixels in the plurality of pixels and the one or more colors. In some embodiments, action 1140 includes determining the number or amount of pixels of the mapped object pixels that reside within each color similarity region for each color, and generating a more favorable (e.g., higher) score for color similarity regions that include more mapped target pixels (e.g., color similarity regions with the most mapped target pixels receive the highest color matching score).

[0170] In some embodiments, action 1140 may include determining a number of valid pixels in a plurality of pixels by comparing each pixel in a plurality of pixels of an object in a multidimensional color space with a plurality of mapped alternative color points corresponding to the color, and / or determining that the object satisfies a minimum color matching threshold for the color based on the number of valid pixels.

[0171] In one or more embodiments, act 1140 may include generating a first color matching score for the object based on determining a distance in the multidimensional color space between each pixel in the plurality of pixels and each mapped color point in a plurality of mapped color points corresponding to the first color. In various embodiments, act 1140 is associated with downsampling pixels in the plurality of pixels before mapping the pixels to the multidimensional color space or generating the one or more color matching scores for the object.

[0172] In some embodiments, action 1140 may include, for each pixel in a plurality of pixels, designating the pixel as valid based on the pixel being within a minimum threshold distance from at least one mapped color point in a plurality of mapped color points corresponding to the first color, and determining that the object matches the first color based on a minimum number of pixels that identify the object that are designated as valid.

[0173] As shown, the series of acts 1100 also includes an act 1150 of classifying the object as a color based on the color matching scores. In particular, act 1150 may involve classifying the object as a first color of the one or more colors based on the one or more color matching scores. In various embodiments, act 1150 may include classifying the object as a color based on a minimum color matching threshold being met for the color.

[0174] The series of actions 1100 may also include a plurality of additional actions. In some embodiments, the series of actions 1100 may include the following actions: receiving a search request for an object having a first color, detecting a plurality of digital images including the object, generating a color match score for the object detected within each of the plurality of digital images with respect to the first color, identifying a subset of the digital images, the subset of the digital images including color match scores that satisfy a minimum color match threshold for the first color, and providing the subset of digital images to a client device associated with the user.

[0175] In various embodiments, the series of acts 1100 may include the acts of detecting multiple instances of an object within a digital image; generating an additional color matching score for each instance of the object; and returning one or more instances of the object having a first color based on the color matching scores.

[0176] In one or more embodiments, the series of actions 1100 may include the following actions: detecting a plurality of object instances of an object in a digital image using a trained object detection neural network, generating a color matching score for each of the plurality of object instances by comparing pixels for each of the plurality of object instances in a multidimensional color space with a plurality of mapped candidate color points corresponding to the color, and determining a subset of the object instances from the plurality of object instances by filtering out object instances that do not meet a minimum color matching threshold from the plurality of object instances.

[0177] In additional embodiments, the series of actions 1100 may include the following actions: identifying a first object instance from the subset of object instances having a highest color match score, determining an additional minimum color match threshold based on the highest color match score of the object instance, and filtering out object instances that do not meet the additional minimum color match threshold from the subset of object instances. In further embodiments, the series of actions 1100 may include an action of providing a digital image having the selected filtered subset of object instances to a client device associated with the user.

[0178] Now turn Figure 12 In one or more embodiments, the series of actions 1200 is implemented on one or more computing devices, such as client devices 102, 700, 800, server device 110, or computing device 1000. Furthermore, in some embodiments, the series of actions 1200 is implemented in a digital environment for creating or editing digital content (e.g., a digital image). In various embodiments, the series of actions 1200 is implemented in a digital environment for detecting instances of objects in a digital image. For example, the series of actions 1200 is implemented on a computing device having a memory that includes a digital image, a selection query including a query color and a corresponding query object, and a trained object mask neural network.

[0179] The series of actions 1200 may include an action 1210 of identifying a query color and a corresponding query object from a query. In some embodiments, action 1210 may involve identifying a query string that includes a query object to be selected in a digital image and a query color corresponding to the query object. In some embodiments, action 1210 further includes analyzing the query string to identify a noun indicating the query object and an adjective indicating the query color.

[0180] As shown, sequence of acts 1200 also includes an act 1220 of mapping the query color to a plurality of points in a color space. Specifically, act 1220 may involve mapping the query color to a plurality of points in a multidimensional color space. In some embodiments, the plurality of points in the multidimensional color space include one or more alternative versions of the query color. In some embodiments, the plurality of points in the multidimensional color space are based on an array of different color brightness levels corresponding to the query color. In various embodiments, act 1220 may include mapping the query color to the plurality of points in the multidimensional color space using a color mapping machine learning model.

[0181] In an example embodiment, act 1220 may include generating an alternative version of the query color by converting a copy of the query color from a first color model corresponding to the multidimensional color space to the alternative color model, modifying one or more color attributes of the query color copy within the alternative color model, and converting the modified query color copy from the alternative color model back to the color model corresponding to the multidimensional color space. In some embodiments, the one or more color attributes of the query color copy within the alternative color model include color brightness, color hue, or color saturation level. In additional embodiments, act 1220 may include modifying the one or more color attributes of the query color copy within the alternative color model by reducing the brightness level of the query color copy within the alternative color model.

[0182] like Figure 12 As shown in , the series of acts 1200 also includes an act 1230 of detecting a query object in the image. In particular, act 1230 may include detecting the query object in the digital image using a trained object detection neural network. In one or more embodiments, act 1230 may include generating an object mask for the query object to encompass pixels of the query object. In some embodiments, act 1230 may also include downsampling the pixels of the query object before generating a color match score for the query object.

[0183] As shown, the series of actions 1200 also includes an action 1240 of generating a color matching score for the query object. Specifically, action 1240 may include generating a color matching score for the query object based on comparing pixels of the query object in a multidimensional color space with a plurality of mapping points corresponding to the query color. In one or more embodiments, action 1240 is based on determining a distance in the multidimensional color space between each pixel of the query object and each of the plurality of mapping points corresponding to the query color.

[0184] In various embodiments, act 1240 includes, for each pixel of the query object, designating the pixel as valid based on the pixel being within a minimum threshold distance from at least one of a plurality of mapping points corresponding to the query color, and determining that the query object matches the query color based on identifying a minimum number of pixels of the query object that are designated as valid.

[0185] As shown, sequence of acts 1200 also includes an act 1250 of providing the image to the user. In particular, act 1250 may include providing the digital image to a client device associated with the user based on determining that the color match score satisfies a minimum color match threshold. In various embodiments, act 1250 may include determining that the query object instance matches the query color based on the color match score satisfying the minimum color match threshold.

[0186] The series of actions 1200 may also include a plurality of additional actions. In one or more embodiments, the series of actions 1200 may include the following actions: detecting a plurality of digital images including a query object; generating a color matching score for the query object detected within each of the plurality of digital images; identifying a subset of digital images including color matching scores that meet a minimum color matching threshold, and providing the subset of digital images to a client device associated with the user.

[0187] In some embodiments, the sequence of actions 1200 may include the following actions: in conjunction with detecting the query object, detecting additional query objects in the image using the trained object detection neural network, generating additional color matching scores for the additional query objects, determining that the additional color matching scores do not satisfy a minimum color matching threshold, and selecting the query object within the image based on the query object satisfying the minimum color matching threshold and the additional color matching scores not satisfying the minimum color matching threshold. In additional embodiments, the sequence of actions 1200 may include providing the digital image by automatically selecting the query object within the digital image using an object mask and / or performing additional operations (e.g., editing the object, such as automatically removing the object).

[0188] In some embodiments, series of actions 1200 may include additional or alternative actions. For example, in some embodiments, action 1200 may include: mapping a query color to a plurality of points in a multidimensional color space; detecting a plurality of instances of a query object in a digital image using a trained object detection neural network; generating a color matching score for each of the plurality of query object instances by comparing pixels of each query object instance in the plurality of query object instances in the multidimensional color space to a plurality of mapped points corresponding to the query color; determining a subset of query object instances from the plurality of query object instances by filtering out query object instances that do not meet a minimum color matching threshold from the plurality of query object instances; and providing a digital image having the selected subset of query object instances to a client device associated with a user.

[0189] In additional embodiments, series of actions 1200 may include identifying a query object instance having a highest color matching score from a subset of query object instances, determining an additional minimum color matching threshold based on the highest color matching score of the query object instance, and filtering out query object instances from the subset of query object instances that do not satisfy the additional minimum color matching threshold.

[0190] As used herein, the term "digital environment" generally refers to an environment implemented, for example, as a standalone application (e.g., a personal computer or mobile application running on a computing device), as an element of an application, as a plug-in to an application, as one or more library functions, as a computing device, and / or as a cloud computing system. The digital medium environment allows a color classification system to detect instances of a detected query object that matches a query color, as described herein.

[0191] Embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory) and executes those instructions, thereby performing one or more processes, including one or more processes described herein.

[0192] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Thus, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0193] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0194] "Network" is defined as one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer properly views the connection as a transmission medium. Transmission media may include networks and / or data links that can be used to carry desired program code devices in the form of computer-executable instructions or data structures and can be accessed by general or special-purpose computers. The above combinations should also be included within the scope of computer-readable media.

[0195] Furthermore, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be automatically transferred from a transmission medium to a non-transitory computer-readable storage medium (device) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to computer system RAM and / or to a less volatile computer storage medium (device) at the computer system. Thus, it should be understood that a non-transitory computer-readable storage medium (device) can be included in computer system components that also (or even primarily) utilize a transmission medium.

[0196] Computer executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a dedicated processing device to perform a specific function or group of functions. In some embodiments, computer executable instructions are executed by a general-purpose computer to convert the general-purpose computer into a special-purpose computer that implements the elements of the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language or even source code. Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as example forms of implementing the claims.

[0197] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment in which local and remote computer systems linked (by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) over a network all perform tasks. In a distributed system environment, program modules can be located in local and remote memory devices.

[0198] Embodiments of the present disclosure may also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources may be rapidly provisioned via virtualization, released with minimal management effort or service provider interaction, and then scaled accordingly.

[0199] The cloud computing model can be composed of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, and the like. The cloud computing model can also disclose various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, and the like. Additionally, as used herein, the term "cloud computing environment" refers to an environment in which cloud computing is employed.

[0200] Figure 13A block diagram of an example computing device 1300 is illustrated, which can be configured to perform one or more of the above-described processes. It will be understood that one or more computing devices, such as computing device 1300, can represent the above-described computing devices (e.g., client devices 102, 700, 800, server device 120, or computing device 1000). In one or more embodiments, computing device 1300 can be a mobile device (e.g., a laptop computer, tablet computer, smartphone, mobile phone, camera, tracker, watch, wearable device, etc.). In some embodiments, computing device 1300 can be a non-mobile device (e.g., a desktop computer, server device, web server, file server, social networking system, program server, application store, or content provider). Additionally, computing device 1300 can be a server device that includes cloud-based processing and storage capabilities.

[0201] like Figure 13 As shown in FIG, computing device 1300 may include one or more processors 1302, memory 1304, storage 1306, input / output ("I / O") interfaces 1308, and communication interfaces 1310, which may be communicatively coupled by way of a communication infrastructure (e.g., bus 1312). Figure 13 The computing device 1300 is shown in FIG. Figure 13 The components illustrated in FIG are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in some embodiments, the computing device 1300 includes fewer than Figure 13 The following will now be described in more detail. Figure 13 Components of computing device 1300 are shown in FIG.

[0202] In certain embodiments, processor 1302 includes hardware for executing instructions, such as those constituting a computer program. By way of example and not limitation, to execute instructions, processor(s) 1302 may retrieve (or fetch) instructions from internal registers, internal cache, memory 1304, or storage 1306, and decode and execute them.

[0203] Computing device 1300 includes memory 1304 coupled to processor(s) 1302. Memory 1304 can be used to store data, metadata, and programs executed by processor(s). Memory 1304 can include one or more of volatile and non-volatile memory, such as random access memory ("RAM"), read-only memory ("ROM"), solid-state disk ("SSD"), flash memory, phase-change memory ("PCM"), or other types of data storage. Memory 1304 can be internal memory or distributed memory.

[0204] Computing device 1300 includes storage 1306, which includes a storage device for storing data or instructions. By way of example and not limitation, storage 1306 may include the non-transitory storage media described above. Storage 1306 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0205] As shown, computing device 1300 includes one or more I / O interfaces 1308, which are provided to allow a user to provide input thereto (e.g., user strokes), receive output therefrom, and otherwise transfer data to or from computing device 1300. These I / O interfaces 1308 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of these I / O interfaces 1308. The touch screen may be activated with a stylus or a finger.

[0206] The I / O interface 1308 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O interface 1308 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0207] The computing device 1300 may also include a communication interface 1310. The communication interface 1310 may include hardware, software, or both. The communication interface 1310 provides one or more interfaces for communicating (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. By way of example and not limitation, the communication interface 1310 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as Wi-Fi. The computing device 1300 may also include a bus 1312. The bus 1312 may include hardware, software, or both that connects the components of the computing device 1300 to each other.

[0208] In the foregoing description, the present invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the present invention have been described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are illustrative of the present invention and should not be construed as limiting the present invention. Numerous specific details have been described to provide a thorough understanding of the various embodiments of the present invention.

[0209] Without departing from the spirit or essential characteristics of the present invention, the present invention may be embodied in other specific forms. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. In addition, the steps / actions described herein may be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes that fall within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to: generating a multidimensional color space for a plurality of colors, the multidimensional color space comprising a plurality of color similarity regions, each color similarity region being associated with a color name, the multidimensional color space comprising a first color similarity region associated with a first color name and a region of the first color within the multidimensional color space, the first color name grouping regions of one or more alternative versions of the first color; identifying an object in a digital image comprising a plurality of pixels; mapping pixels of the plurality of pixels to the multidimensional color space to determine one or more color correspondences with one or more color similarity regions of the plurality of color similarity regions, the one or more color correspondences corresponding to one or more color names; generating one or more color matching scores for the object based on the one or more color correspondences between the pixels in the plurality of pixels and the one or more color similarity regions; as well as The object is classified as the first color having a first color name among the plurality of colors based on the one or more color matching scores.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: downsample the plurality of pixels before generating the one or more color matching scores for the object.

3. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate a second color similarity region corresponding to the second color by grouping the second color and one or more additional alternative versions of the second color together within the multidimensional color space.

4. The non-transitory computer-readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate an alternative version of the first color by: converting the first color from a first color model corresponding to the multidimensional color space to a second color model; modifying one or more color properties of the first color within the second color model; and The first color having the modified one or more color properties is converted from the second color model back to the first color model corresponding to the multidimensional color space. 5 . The non-transitory computer-readable medium of claim 4 , wherein the one or more color properties of the first color that are modified within the second color model include color brightness, color hue, or color saturation level.

6. The non-transitory computer-readable medium of claim 4, wherein the instructions, when executed by the at least one processor, cause the computing device to modify the one or more color attributes of the first color within the second color model by reducing a brightness level of a copy of the first color in the second color model.

7. The non-transitory computer-readable medium of claim 3, wherein the first color similarity region for the first color comprises: A first color point for the first color and a plurality of mapped candidate color points for the first color mapped to the multi-dimensional color space.

8. The non-transitory computer-readable medium of claim 7 , wherein the instructions, when executed by the at least one processor, cause the computing device to: generate a first color matching score for the object based on determining a distance in the multidimensional color space between each of the plurality of pixels of the object and the first color point and each of the plurality of mapped alternative color points.

9. The non-transitory computer-readable medium of claim 7, further comprising instructions that, when executed by the at least one processor, cause the computing device to: designating the pixel as valid based on the pixel being within a minimum threshold distance from at least one of the plurality of mapped color points corresponding to the first color or from the first color point; and The object is determined to match the first color based on a minimum percentage of pixels designated as valid that identifies the object.

10. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: receiving a search request for the object having the first color; detecting a plurality of digital images including the object; generating a color matching score for the first color for the object detected within each of the plurality of digital images; identifying a subset of digital images in the plurality of digital images that includes the object having a color matching score that satisfies a minimum color matching threshold for the first color; and In response to the search request, the subset of digital images is provided.

11. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: detecting a plurality of instances of the object within the digital image; generating a color matching score for each instance of the object; classifying one or more instances of the object as the first color based on the color matching score; and The one or more instances of the object having the first color are returned.

12. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: receiving a query request for the object indicating the first color by a color name; In response to the query request indicating the color name of the first color, identifying a color similarity region within the multi-dimensional color space for the first color that is pre-mapped to the first color name; and In response to identifying the color similarity region for a first color within the multidimensional color space, an indication that an object in the digital image matches the first color is returned based on the object being classified as the first color.

13. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to: identify the object in the digital image using an object detection neural network.

14. A system for classifying objects in a digital image based on color, the system comprising: one or more memory devices, the one or more memory devices comprising a multidimensional color space for a plurality of colors, the multidimensional color space comprising a plurality of color similarity regions, each color similarity region being associated with a color label, the multidimensional color space comprising a first color similarity region, the first color similarity region being associated with a first color label and a region of a first color within the multidimensional color space, the first color label grouping regions of one or more alternative versions of the first color; as well as at least one server device, the at least one server device being configured to cause the system to: identifying an object in a digital image comprising a plurality of pixels; mapping pixels of the plurality of pixels to the multidimensional color space to determine a correspondence with one or more color similarity regions of the plurality of color similarity regions, the correspondence corresponding to one or more color labels; generating one or more color matching scores for the object based on the one or more color correspondences between the pixels in the plurality of pixels and the one or more color similarity regions; as well as The object is classified as the first color of the plurality of colors having a first color label based on the one or more color matching scores.

15. The system of claim 14, wherein the at least one server device is further configured to generate a first color similarity region for the first color by grouping the first color and one or more alternative versions of the first color together in the multidimensional color space.

16. The system of claim 15, wherein the at least one server device is further configured to cause the system to generate an alternative version of the first color by: converting the first color from a first color model corresponding to the multidimensional color space to a second color model; modifying one or more color properties of the first color within the second color model; and The first color having the modified one or more color properties is converted from the second color model back to the first color model corresponding to the multidimensional color space.

17. The system of claim 16, wherein the first color similarity region for the first color comprises: A first color point for the first color and a plurality of mapped candidate color points for the first color mapped to the multi-dimensional color space.

18. The system of claim 17 , wherein the at least one server device is further configured to cause the system to: generate a first color matching score for the object based on determining a distance in the multidimensional color space between each of the plurality of pixels of the object and the first color point and each of the plurality of alternative mapped color points.

19. A computer-implemented method for classifying an object into colors, comprising: generating a multidimensional color space for a plurality of colors, the multidimensional color space comprising a plurality of color similarity regions, each color similarity region being associated with a color name, the multidimensional color space comprising a first color similarity region associated with a first color name and a region of the first color within the multidimensional color space, the first color name grouping regions of one or more alternative versions of the first color; identifying a first object comprising a first plurality of pixels in a digital image; identifying a second object in the digital image comprising a second plurality of pixels having different color values than the first plurality of pixels; mapping pixels of the first plurality of pixels to the multidimensional color space to determine a first set of color correspondences with a first set of color similarity regions among the plurality of color similarity regions, the first set of color correspondences corresponding to a first set of one or more color names; mapping pixels of the second plurality of pixels to the multidimensional color space to determine a second set of color correspondences with a second set of color similarity regions of the plurality of color similarity regions, the second set of color correspondences corresponding to a second set of one or more color names; generating a first set of color matching scores for the first object based on the first set of color correspondences between the first plurality of pixels and the first set of color similarity regions; generating a second set of color matching scores for the second object based on the second set of color correspondences between the second plurality of pixels and the second set of color similarity regions; classifying the first object as the first color having a first color name among the plurality of colors based on the first set of color matching scores; and The second object is classified as the first color having the first color name among the plurality of colors based on the second set of color matching scores.

20. The computer-implemented method of claim 19, further comprising downsampling the first plurality of pixels and the second plurality of pixels before generating the first set of color matching scores and the second set of color matching scores for the first object and the second object.

Images