Identify visually similar digital images based on user-selected visual attributes

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media for identifying similar digital images for a query digital image using a deep neural network-based model. For example, the disclosed system uses a deep neural network-based model to analyze a query digital image to generate a deep neural network-based representation of the query digital image. In addition, the disclosed system can generate results of visually similar digital images for the query digital image based on comparing the deep neural network-based representation with a representation of a candidate digital image. In addition, the disclosed system can identify visually similar digital images based on user-defined attributes and image masks to emphasize specific attributes or portions of the query digital image.

Description

Background Art

[0001] Advances in computing devices and image analysis techniques have led to various innovations in identifying visually similar digital images. For example, image analysis systems are now able to analyze high-resolution digital images to identify objects within the images and search terabytes of information stored in digital image databases to identify other digital images depicting the same or similar objects.

[0002] However, despite these advances, conventional image analysis systems still suffer from a number of shortcomings, particularly in terms of accuracy and flexibility in identifying similar digital images. For example, while conventional image analysis systems can identify the same object in two different digital images, these systems typically ignore other aspects of the image (e.g., background, spatial arrangement of objects, and other visual attributes of the image). In fact, because conventional image analysis systems typically rely solely on semantic content to classify images based on various image tags, these systems place too much emphasis on objects in their analysis. As a result, conventional image analysis systems often produce inaccurate results when determining the visual similarity of two images. This is a particularly important issue because, due to this inaccuracy, users of conventional image analysis systems often need to spend too much time performing too many user actions to search for matching results before locating the desired image match.

[0003] In addition, conventional image analysis systems are generally inflexible. Indeed, as described above, conventional image analysis systems are generally one-dimensional in that they match digital images based solely on identifying specific objects within the images. Many of these systems also require very specific inputs (e.g., a single digital image) to be used as a basis for finding matching images. In situations where a user wants to find similar images for more than one input image, many conventional image analysis systems need to perform multiple single image searches and / or retrain the analysis model to accommodate multiple input images. Due to their inflexible nature, these conventional systems are generally unable to customize images that match the needs of the user, other than searching for a single image of a specific object.

[0004] Thus, there are several disadvantages with conventional image analysis systems. Summary of the invention

[0005] By providing systems, methods and non-transitory computer-readable media that accurately and flexibly identify digital images that share visual attributes using models based on deep neural networks, the embodiments described herein provide benefits and solve one or more of the foregoing or other problems in the art. For example, the disclosed system can match digital images based on at least visual attributes such as spatial selectivity, image synthesis and / or object counting. In addition, the disclosed system can identify digital image matches based on a synthetic analysis of visual attributes from multiple digital images. Although the present disclosure relates to a system for simplification, the features outlined herein may also be attributed to the various methods and non-transitory computer-readable media of the present disclosure.

[0006] To illustrate, in some embodiments, the disclosed system receives a user selection of a query digital image and at least one of spatial selectivity, image synthesis, and / or object counting for identifying visually similar digital images. The system further utilizes the trained deep neural network-based model to generate a deep neural network-based representation of the query digital image. Based on the deep neural network-based representation of the query digital image, the disclosed system identifies similar digital images to the query digital image from a digital image database.

[0007] In some embodiments, the disclosed system receives a multi-image query identifying multiple query images, and then identifies digital images that are similar to the combined visual attributes of the multiple query images. In some embodiments, the disclosed system facilitates multi-query similarity by generating a composite feature vector representing the visual attributes of the multiple query images. The disclosed system utilizes a trained deep neural network-based model to identify similar digital images based on the composite feature vector.

[0008] The disclosed systems, methods, and non-transitory computer-readable media provide several advantages over conventional image analysis systems. For example, the disclosed systems are capable of determining image similarity based on multiple visual attributes such as spatial selectivity, image composition / layout, and object count. As a result, the disclosed systems identify images that are visually similar to a query digital image. Furthermore, because the disclosed systems are capable of receiving indications of visual attributes of one or more query digital images, the disclosed systems can flexibly identify matching digital images based on a variety of user-selected factors (e.g., multiple query digital images and various visual attributes).

[0009] Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure will describe one or more embodiments of the present invention with additional particularity and detail by referring to the accompanying drawings. The following paragraphs briefly describe these drawings, in which:

[0011] Figure 1 shows a user interface including querying a digital image according to one or more embodiments;

[0012] Figure 2 illustrates a user interface including a query digital image with an image mask according to one or more embodiments;

[0013] Figure 3 illustrates a user interface including a plurality of query digital images according to one or more embodiments;

[0014] Figure 4 illustrates a user interface including a plurality of image masks according to one or more embodiments;

[0015] Figure 5 shows a modification result of a similar digital image based on receiving a new image mask according to one or more embodiments;

[0016] Figure 6 illustrates a user interface for identifying similar digital images based on object counts in accordance with one or more embodiments;

[0017] Figure 7 Training a deep neural network based model according to one or more embodiments is shown;

[0018] Figure 8 A schematic diagram of a digital image matching system according to one or more embodiments is shown;

[0019] Fig. 9 An example environment for implementing a digital image matching system according to one or more embodiments is shown;

[0020] Fig.10 A flow diagram illustrating a series of actions for identifying similar digital images to a query digital image in accordance with one or more embodiments.

[0021] Fig.11 shows a series of actions in a step for identifying similar digital images to a query digital image according to one or more embodiments;

[0022] Fig.12 shows a series of actions in steps for identifying similar digital images of composite feature vectors according to one or more embodiments; and

[0023] Fig.13 A block diagram of an example computing device is shown in accordance with one or more embodiments. DETAILED DESCRIPTION

[0024] One or more embodiments described herein provide benefits and solve one or more of the foregoing or other problems in the art by providing a digital image matching system that utilizes a deep neural network-based model to accurately and flexibly identify digital images that share visual attributes. For example, the digital image matching system can match digital images based on spatial selectivity, image synthesis, and / or object counting. In addition, the digital image matching system can identify similar digital images based on a composite analysis of visual attributes from multiple query images.

[0025] To illustrate, in some embodiments, in addition to a user's selection of at least one of spatial selectivity, image composition, and / or object count for a query digital image, the digital image matching system receives a query digital image for identifying visually similar digital images. The digital image matching system also generates a deep neural network-based representation (e.g., a feature vector) of the query digital image using a trained deep neural network-based model. The digital image matching system may also determine similarity scores for a plurality of candidate digital images stored in a digital image database by comparing features of the query digital image (e.g., a deep neural network-based representation) with features of stored digital images. Based on the similarity scores, the digital image matching system identifies similar digital images for the query digital image from the digital image database.

[0026] In some embodiments, a digital image matching system receives more than one query digital image, each query digital image including an indication of visual attributes for identifying digital images that share the indicated attributes. In these embodiments, the digital image matching system generates multiple query vector representations representing features of each query digital image. Based on the multiple query vector representations, the digital image matching system identifies similar digital images from a digital image database.

[0027] To generate a deep neural network-based representation for a digital image, in some embodiments, the digital image matching system trains a deep neural network-based model to predict features for a training digital image and compares the predicted features to actual (e.g., ground truth) features. In practice, the digital image matching system utilizes a set of training data, such as a plurality of training images and corresponding ground truth feature vectors, to train the deep neural network-based model to correctly predict feature vectors of a digital image.

[0028] In some embodiments, as part of generating features for comparison between a query digital image and stored candidate digital images, a digital image matching system may utilize a trained neural network (e.g., a deep neural network-based model) to predict scores for a number of labels that cover a wide variety of textual concepts of objects that may be depicted in an image (e.g., car, person, tree, sun, etc.). The digital image matching system utilizes a set of training data, such as a plurality of training images and corresponding training labels, to train a deep neural network-based model to correctly predict scores for various labels. In some embodiments, the digital image matching system may predict scores for a set of image labels associated with a deep neural network-based model. In practice, the trained deep neural network-based model may predict or determine a score for each image label (e.g., car, person, tree, sun, etc.) of the query digital image to determine, for each image label, the probability that the query digital image contains or depicts an object corresponding to the given image label.

[0029] As described above, the digital image matching system receives a query digital image. Specifically, the digital image matching system receives the query digital image from a user client device. In addition, in some embodiments, the digital image matching system receives a user selection of visual attributes in conjunction with the query digital image for identifying visually similar digital images. For example, the digital image matching system receives a user selection of one or more of spatial selectivity, image synthesis, and / or object counting. Additionally or alternatively, the digital image matching system receives a user selection of a query digital image received from another source and / or stored in a digital image database associated with the digital image matching system.

[0030] As a baseline, the digital image matching system utilizes a deep neural network-based model to generate a feature map of a query digital image. Based on the generated feature map, the digital image matching system also utilizes one or more of a spatially selective algorithm, an image composition algorithm, and / or an object counting algorithm to generate a deep neural network-based representation of the query digital image. To illustrate, in response to any user selection in conjunction with visual attributes to serve as a basis for identifying similar digital images and receiving a query digital image, the digital image matching system can extract a feature map of the query digital image, and by using one or more of the spatially selective algorithm, the image composition algorithm, and / or the object counting algorithm, a set or sequence of deep features can be generated from the feature map representing the query digital image.

[0031] As described above, the digital image matching system analyzes the query digital image based on the selected visual attributes (e.g., semantic content, spatial selectivity, image synthesis, and / or object count) of the query digital image. For example, in order to generate a deep neural network-based representation of the query digital image, the digital image matching system can utilize a spatial selectivity algorithm based on receiving user selections to match digital images based on spatial selectivity. Similarly, the digital image matching system can utilize an image synthesis algorithm based on receiving user selections to match digital images based on image synthesis. Additionally or alternatively, the digital image matching system can use an object counting algorithm based on receiving user selections to match digital images based on object counts. In some embodiments, the digital image matching system can utilize a hybrid or composite algorithm to match digital images based on two or more of spatial selectivity, image synthesis, and object counts.

[0032] In addition to or as part of generating the deep neural network-based representation, the digital image matching system can apply weights to certain indicated regions or portions of the query digital image. In practice, the digital image matching system can receive an image mask that highlights, delineates, or otherwise defines one or more portions of the query digital image, and the digital image matching system can modify the deep neural network-based representation accordingly to apply weights to the regions indicated by the image mask so that features corresponding to the identified portions of the image have a greater influence on matching with features of the stored digital image. By modifying the deep neural network-based representation to apply weights to masked regions / portions, the digital image matching system can emphasize user-indicated portions of the query digital image to identify digital images that more closely resemble the masked portions of the query digital image.

[0033] As described above, in some embodiments, the digital image matching system receives more than one query digital image. In these embodiments, the digital image matching system generates multiple query vector representations representing visual attributes of one or more received query digital images. For example, the digital image matching system generates a composite feature vector using a trained deep neural network-based model. In addition, the digital image matching system identifies similar digital images by identifying digital images having features that are the same or similar to the features of the composite feature vector.

[0034] In a multi-query scenario where a digital image matching system receives two or more query digital images to be considered together in identifying similar digital images, the digital image matching system may also receive a user selection of visual attributes (e.g., spatial selectivity, image synthesis, and / or object count). Thus, the digital image matching system may generate or modify a multi-query vector representation of the constituent digital images based on receiving the user selection of the visual attributes for use in identifying similar digital images. For example, the digital image matching system may modify the multi-query vector representation using one or more of a spatial selectivity algorithm, an image synthesis algorithm, and / or an object counting algorithm.

[0035] In addition, the digital image matching system can also generate or modify multiple query vector representations based on receiving image masks of multiple query digital images associated with the composite feature vector. Therefore, as mentioned above with respect to a single query digital image, the digital image matching system can more heavily weight or emphasize those portions of the query digital image that are indicated by the image mask to identify similar digital images.

[0036] In addition, the digital image matching system can identify similar digital images for the query digital image from the digital image database. Specifically, the digital image matching system can determine the similarity scores of multiple digital images within the digital image database. For example, the digital image matching system can determine the similarity score by comparing the features of the query digital image (e.g., features within a representation based on a deep neural network) with the features of the candidate digital image. In some embodiments, the digital image matching system can compare the image label probability of each of the multiple digital images with the image label probability of the query digital image (or composite feature vector). In addition, the digital image matching system can rank the multiple digital images based on the corresponding similarity scores. Therefore, in order to identify similar (e.g., matching) digital images, the digital image matching system can provide multiple top-ranked digital images, or can identify digital images that meet a specific similarity threshold.

[0037] Digital image matching systems provide several advantages over traditional image analysis systems. For example, digital image matching systems improve the accuracy and effectiveness of traditional image analysis systems. To illustrate, because digital image matching systems utilize multiple visual attributes and / or multiple digital images to identify matching digital images, digital image matching systems more accurately identify visually similar images than some traditional image analysis systems that rely only on semantic content analysis and use only a single digital image as a query.

[0038] As a result of the improved accuracy of the digital image matching system, the digital image matching system further increases the speed and reduces the number of actions required for a user to identify a desired image match. For example, a user of a conventional image analysis system may spend an excessive amount of time performing numerous scrolling and / or other navigation operations, while on the other hand, a user associated with the disclosed digital image matching system performs fewer actions and spends less time because the digital image matching system generates and provides better image matches.

[0039] In addition, the digital image matching system also improves flexibility. For example, although some traditional image analysis systems rely only on semantic visual attributes to identify objects within an image, the digital image matching system adapts to more customized image matching. In fact, in addition to other visual attributes such as spatial selectivity, image synthesis, and object counting, the digital image matching system can also identify matching digital images based on semantic visual attributes. In some embodiments, the digital image matching system can receive (for example, from a user client device) an indication of which visual attribute (of one or more query digital images) is used as the basis for identifying matching digital images, thereby allowing the user to customize how the digital image matching system identifies matches. The digital image matching system also provides more flexible matching by enabling users to define image masks to emphasize that portions of one or more query digital images match other digital images.

[0040] The digital image matching system also performs faster than many traditional image analysis systems. Specifically, while some traditional systems require retraining the model each time a user wants to match a digital image based on different visual attributes, the disclosed digital image matching system utilizes a deep neural network-based model that does not require such repeated retraining. Instead, the digital image matching system generates a deep neural network-based representation that considers multiple visual attributes simultaneously, thereby saving processing time and computing power that might otherwise be used to repeatedly train the model. In addition, the digital image matching system implements a multi-query technique that enables the digital image matching system to analyze multiple digital images in succession to identify a merged digital image that is similar to features of multiple images. Therefore, the digital image matching system is computationally faster and more efficient than traditional systems that, given multiple query digital images, require multiple single query analyses and comparison of the results of each analysis to identify digital images similar to each query digital picture.

[0041] As shown in the previous discussion, the present disclosure utilizes various terms to describe the features and benefits of the digital image matching system. Hereafter, additional details are provided regarding the meaning of these terms used in the present disclosure. For example, as used herein, the term "query digital image" refers to a digital image that the digital image matching system uses as a basis for finding similar digital images. A query digital image may refer to a digital image that a user uses as a search query to find visually similar digital images. In practice, a query digital image may refer to a digital image provided by a user and received by the digital image matching system from a user client device. For example, a query digital image may refer to a digital image that a digital image matching system analyzes using a deep neural network-based model to generate a deep neural network-based representation. In practice, in some embodiments, the digital image matching system extracts a deep neural network-based representation of the query digital image in the form of a feature vector.

[0042] As described above, the digital image matching system analyzes and classifies the query digital image to identify similar digital images. As used herein, the term "similar digital image" refers to a digital image that is similar to the query digital image (e.g., similar in appearance and / or in terms of unobservable deep features). Specifically, a similar digital image may refer to a digital image that shares visual attributes with the query digital image. For example, the digital image matching system may identify (e.g., match) similar digital images as digital images having a similarity score that satisfies a similarity threshold. Similar digital images may correspond to a set of deep features that the digital image matching system identifies as being similar to a deep neural network-based representation of the query digital image.

[0043] As described above, the digital image matching system receives user input to set or determine visual attributes to use as a basis for identifying similar digital images. As used herein, the term "visual attribute" refers to an attribute or characteristic of a digital image that is visible to an observer. Visual attributes may refer to attributes that the digital image matching system determines or identifies by analyzing the digital image (e.g., through a deep neural network-based model or other means). For example, visual attributes may refer to one or more of semantic content, spatial selectivity, image synthesis, and / or object counts. Although visual attributes correspond to observable characteristics of an image, visual attributes may be represented or defined by unobservable characteristics of an image, such as depth features.

[0044] As used herein, the term "semantic content" refers to a type of visual attribute of a digital image. Specifically, semantic content may refer to image labels that describe various attributes of a digital image, such as objects contained or depicted in the digital image. To illustrate, a digital image matching system may utilize a convolutional neural network trained on a large-scale image classification dataset (e.g., ImageNet) to determine semantic content by predicting a score for a set of image labels - that is, by determining the probability that a given digital image depicts the object described by the image label.

[0045] As used herein, the term "image tag" refers to a text-based descriptor of a digital image. Specifically, an image tag may refer to a text string that describes one of a variety of text concepts that a digital image may depict. These text concepts may include, but are not necessarily limited to, specific object names, object descriptions, number of objects, landscape descriptions, geographic locations of scenes within a digital image, and locations of objects within a digital image, etc.

[0046] As described above, the digital image matching system can determine visual attributes such as spatial selectivity. As used herein, the term "spatial selectivity" refers to a location within a digital image. For illustration, spatial selectivity can include a coordinate position or spatial position index in the form of a vertical (e.g., row) coordinate i and a horizontal (e.g., column) coordinate j. Therefore, spatial selectivity can refer to the coordinate position of an object within a digital image. As described in further detail below, the digital image matching system can utilize a "spatial selectivity algorithm" to generate a deep neural network-based representation of a query digital image based on a weighted global average pooling ("pooling") of the positions of the determined digital images. For example, using spatial selectivity, the digital image matching system can identify images that not only depict the same object as the query digital image, but also depict objects in similar positions to the query digital image.

[0047] The digital image matching system may also determine an image composition of the digital image. As used herein, the term "image composition" refers to the arrangement or layout of a digital image. Specifically, image composition may refer to the relational positions of objects within a digital image (e.g., relative to other objects). As described in further detail below, the digital image matching system may utilize an "image composition algorithm" to generate a deep neural network-based representation of a query digital image based on establishing positional similarities of spatial arrangements of digital images. For example, using composition, the digital image matching system may identify images that not only depict the same objects as the query digital image, but also depict objects in a similar arrangement as the query digital image.

[0048] In addition, the digital image matching system can determine an object count of the digital image. As used herein, the term "object count" refers to the number of objects within a digital image. Specifically, the object count can refer to a number of objects within a particular location or portion of a digital image. As described in further detail below, the digital image matching system can utilize an "object counting algorithm" based on implementing a "subitizing-based" similarity search to generate a deep neural network-based representation of a query digital image to identify digital images with similar object counts as the query digital image.

[0049] As described above, the digital image matching system can generate a deep neural network-based representation for a query digital image. As used herein, the term "deep neural network-based representation" refers to a feature vector representation of a digital image. In some embodiments, the deep neural network-based representation can refer to a set or sequence of deep features representing a digital image. For example, the digital image matching system can generate a deep neural network-based representation by utilizing at least one of a spatially selective algorithm, an image synthesis algorithm, or an object counting algorithm, as described above and described in further detail below.

[0050] As used herein, the term "deep features" (or sometimes simply "features") refers to features of a digital image. Deep features may include visible features as well as latent or hidden features of a digital image analyzed within a neural network. To illustrate, a deep feature may include one or more feature vectors generated by one or more layers of a neural network to reflect a digital image. A deep feature may include, for example, characteristics of a digital image at different levels of abstraction generated at various layers of a neural network. A deep feature may include non-linear characteristics of a digital image that are not interpretable to a human viewer.

[0051] As used herein, the term "deep neural network-based model" refers to a machine learning model. Specifically, a deep neural network-based model may refer to a machine learning model, which is composed of one or more deep neural networks, and the digital image matching system uses the one or more neural networks to generate a feature map of a digital image and / or a deep neural network-based representation. As used herein, the term "neural network" (or "artificial neural network") refers to an artificial machine learning model that learns a task by analyzing example data using multiple layers, and the task analyzes the features of example data at different abstract levels. Specifically, the term "neural network" may include a model of interconnected digital neurons that transmits and learns to approximate complex functions and generates outputs based on multiple inputs provided to the model. For example, a deep neural network may include an output layer, such as a softmax layer, with M hidden layers between the softmax layer and the input layer. Neural networks such as convolutional neural networks can model complex nonlinear relationships by generating a synthetic model that represents data as a hierarchical synthesis of information.

[0052] As described above, the digital image matching system utilizes a trained deep neural network-based model to generate a deep neural network-based representation of a query digital image. As used herein, the term "training" or "trained" or "in training" refers to the process of teaching or adjusting a machine learning model (e.g., a deep neural network-based model). Specifically, training can refer to the process of increasing the accuracy of a machine learning model using training data to generate a representative feature vector.

[0053] As further mentioned, the digital image matching system may receive an image mask for a digital query image. As used herein, the term "image mask" refers to an indication of a portion of a digital image. Specifically, the image mask may refer to an area or portion of a digital query image that is selected, highlighted, outlined, or otherwise defined by a user through a user interface on a user client device. Based on receiving the image mask, the digital image matching system may modify the deep neural network-based representation to weight any portion of the digital query image indicated by the image mask.

[0054] As described above, the digital image matching system may receive multiple query digital images from a user client device, and may generate a multi-query vector representation to serve as a basis for identifying similar digital images (e.g., images that share visual attributes with each of the received query digital images). As used herein, the term "multi-query vector representation" refers to a representation based on a deep neural network. The multi-query vector representation may refer to a set of deep features generated by the digital image matching system to represent more than one query digital image together. In some embodiments, the digital image matching system may modify the deep neural network-based representation by utilizing aggregation techniques to generate a multi-query vector representation. For example, the digital image matching system may generate a multi-query vector representation in the form of a composite feature vector. As used herein, the term "composite feature vector" refers to a feature vector that includes features (e.g., deep features) of multiple digital images. Therefore, the composite feature vector represents a synthesis or merging of multiple visual attributes that make up the query digital image.

[0055] More details about the digital image matching system will now be provided with reference to the accompanying drawings. Figure 1 A user interface 100 is shown that is displayed to a user via a user client device. Among other features, Figure 1 The user interface 100 also includes a query digital image 102 , identified similar digital images 104 , a visual attribute option 106 , and a weighting option 108 .

[0056] As described above, the digital image matching system provides a user interface 100 to a user client device. As also mentioned, based on receiving a query digital image 102 from a client device, the digital image matching system provides an identified similar digital image 104 to the user client device. In order to identify and provide the similar digital image 104, the digital image matching system analyzes the query digital image 102. For example, the digital image matching system utilizes one or more neural networks (e.g., convolutional neural networks) to generate deep feature sequences of the query digital image 102 and candidate digital images stored in a database.

[0057] More specifically, the digital image matching system utilizes a deep neural network based model to generate a feature map (e.g., as an output of a pool 5 layer) containing features of the query digital image 102. Based on the feature map, the digital image matching system further generates a deep neural network based representation of the query digital image 102 by aggregating the feature maps according to one or more of spatial selectivity, image synthesis, or object counting.

[0058] In addition, the digital image matching system may rank the plurality of digital images within the digital image database in order of relevance with respect to the query digital image 102. For example, the digital image matching system compares the candidate digital images with the query digital image 102 to determine which images are most similar. In practice, the digital image matching system may compare the candidate digital images with the query digital image 102 by utilizing a neural network to generate deep features of the candidate digital images, and by further comparing the generated feature vectors of the candidate digital images with the deep neural network-based representation of the query digital image 102. In addition, the digital image matching system provides similar digital images 104 to the user client device in order of relevance to the query digital image 102.

[0059] In some embodiments, the digital image matching system determines a similarity score for each of the plurality of digital images and ranks the images based on their respective similarity scores. For example, the digital image matching system may utilize a similarity search function to generate the similarity scores. In some embodiments, the digital image matching system may utilize different similarity search functions for spatial selectivity, image synthesis, and object counting, as described in further detail below. In practice, the digital image matching system may modify the similarity search function based on different considerations for identifying similar images based on spatial selectivity, image synthesis, and object counting.

[0060] In some embodiments, the digital image matching system may determine the similarity score by utilizing a loss function. In practice, the digital image matching system may utilize a loss function to compare the deep neural network-based representation of the query digital image 102 with the feature vectors of the candidate digital images. Therefore, the digital image matching system may utilize a loss function (e.g., a mean square error function or a cross entropy loss function) to determine a similarity score in the form of a loss metric between the features of the query digital image 102 and the features of the candidate images. In some embodiments, the digital image matching system normalizes the similarity score (e.g., on a scale from 0 to 1). In some embodiments, a given similarity score may be directly related to a determined loss metric, while in other embodiments, the similarity score may be inversely related to a determined loss metric.

[0061] In some embodiments, the digital image matching system determines the semantic similarity score by comparing the image tag score of the query digital image 102 with the image tag score of each of the plurality of digital images. In practice, the digital image matching system may determine the image tag score of the query digital image 102 based on a deep neural network-based representation of the query digital image 102. Similarly, for the candidate digital images, the digital image matching system may determine (or access predetermined) image tag scores of the candidate digital images based on feature vectors associated with each candidate image.

[0062] In order to compare the image tag score of the query digital image 102 with the image tag scores of the candidate digital images, the digital image matching system may determine the difference between the image tag scores. For example, the digital image matching system may compare the amount of similarity by determining the difference between the score of the first image tag of the candidate digital image and the score of the same image tag of the query digital image 102. Similarly, the digital image matching system may determine the difference between each image tag score for the query digital image 102 and the candidate digital images. In some embodiments, the digital image matching system may utilize a distance function to determine the total (e.g., average) distance between all (or a subset) of the image tag scores of the query digital image 102 and a given candidate digital image. The digital image matching system determines higher similarity scores for those digital images whose image tag scores are more closely similar to the image tag score associated with the query digital image 102 (e.g., the image tag scores have a smaller distance or difference between them).

[0063] To elaborate, as a baseline for generating a deep neural network-based representation of a query digital image, the digital image matching system utilizes a residual network model (e.g., TagRestNet-50) that has been trained for image tagging—i.e., for determining the probability that each of a set of image tags corresponds to a query digital image 102. Given the query digital image 102, the digital image matching system utilizes the deep neural network-based model to predict or determine a score for a set of image tags covering a variety of ranges of textual concepts (e.g., a set of 18,157 image tags). The digital image matching system performs a forward pass of the deep neural network-based model for the query digital image 102 to extract feature maps as a resource for generating a deep neural network-based representation of the query digital image 102. For example. From the deep neural network-based model, the digital image matching system obtains the output of the Pool 5 layer As a feature of searching and identifying similar digital images 104, and They are feature maps The number, height and width of channels.

[0064] Digital image matching systems aggregate feature maps globally And reshape the feature map into a feature vector In addition, in order to search for similar digital images, the digital image matching system uses a similarity search function F(·), which converts the convolutional neural network feature vector y of the query digital image into gp More specifically, the digital image matching system implements a similarity search operation S = F (y gp ), which queries the feature vector y of the digital image 102 gp Takes as input and returns a list of similarity scores for multiple digital images from a digital image database Where N is the number of digital images in the digital image database.

[0065] For efficient, non-exhaustive search of digital image databases to identify similar digital images, the digital image matching system can further utilize optimized product quantization ("OPQ") technology and reverse indexing technology. For example, the digital image matching system can implement the OPQ technology described in TiezhengGe, Kaiming He, Qifa Ke and Jian Sun in Optimized Product Quantization for Approximate Nearest Neighbor Search, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2946-53 (2013), the entire contents of which are incorporated herein by reference. Specifically, the digital image matching system can decompose the high-dimensional space into the Cartesian product of a finite number of low-dimensional subspaces, and the finite number of low-dimensional subspaces are each quantized separately. In addition, the digital image matching system can optimize product quantization by minimizing quantization distortion by applying non-parametric methods and / or parametric methods that achieve the optimal solution when the input data follows a Gaussian distribution.

[0066] like Figure 1As shown, the user interface 100 includes an indication of a user selection of a visual attribute option 106 for identifying similar digital images 104. Based on the user selection of a particular visual attribute option 106, the digital image matching system generates a corresponding deep neural network-based representation to identify similar digital images 104. Specifically, the user interface 100 shows that the user has selected to search for similar digital images based on "similarity." Based on receiving a query digital image 102 in combination with the user's selection of the visual attribute "similarity," the digital image matching system generates a deep neural network-based representation for the query digital image 102 as a basis for identifying similar digital images 104 according to spatial selectivity constraints. Specifically, the digital image matching system utilizes a spatially selective algorithm based on the user selection of "similarity." Although Figure 1 A selectable option labeled "Similarity" is shown, but in some embodiments, user interface 100 includes a different label, such as "Selective" or "Spatially Selective" or a similar label.

[0067] In any case, based on the user selection of the visual attribute option 106 corresponding to the spatially selective analysis, the digital image matching system utilizes the spatially selective algorithm to generate a deep neural network-based representation for the query digital image 102. More specifically, the digital image matching system utilizes the feature map of the above baseline To implement the position-weighted global pooling technique. To illustrate, as mentioned above, the digital image matching system uses a deep neural network-based model to extract feature maps And the digital image matching system uses these feature maps as input to the global average aggregation function. For example, the digital image matching system determines the aggregated feature map (for the query digital image 102) by the following formula:

[0068]

[0069] where i and j are spatial position indices in the vertical / row and horizontal / column directions, respectively. Therefore, digital image matching systems aggregate feature maps according to spatially selective algorithms. Generate the feature vector y gp A deep neural network-based representation of a query digital image 102 in the form of .

[0070] By utilizing the spatially selective algorithm, the digital image matching system treats all locations within the query digital image 102 equally, ignoring the importance of certain sub-regions in the feature map and the query digital image 102. If the digital query image contains a small-sized object of interest surrounded by a larger background region, however, the global average pooling technique of the spatially selective algorithm may bias the results (e.g., feature vectors) toward the background region as a result of the equal processing on the query digital image 102.

[0071] Thus, the digital image matching system identifies similar digital images 104 that are more visually similar to background areas than to foreground objects. Figure 1 As shown, the digital image matching system identifies similar digital images 104 that are visually similar to the query digital image 102 based on an equally weighted analysis on the query digital image 102. As described above, the digital image matching system can identify similar images based on a comparison of deep features. In practice, the digital image matching system can determine and / or access a deep neural network-based representation of the digital image to compare with the deep neural network-based representation of the query digital image 102.

[0072] In some embodiments, the digital image matching system implements a position-weighted global average pooling technique to weight certain portions of the query digital image 102 more heavily (e.g., to emphasize those portions when identifying similar digital images). By weighting portions of the query digital image 102, the digital image matching system can identify similar digital images that are different from the similar digital images 104 by biasing the search toward digital images that are more similar in visual attributes to any indicated portion of the query digital image 102. In effect, Figure 2 A user interface 100 is shown including a query digital image 102 and identified similar digital images 202. As shown, the similar digital images 202 are Figure 1 The similar digital images 104 in are different. Specifically, Figure 1 The similar digital images 104 in FIG. 1 each show a bicycle (most also showing a person accompanying the bicycle), while Figure 2 Similar digital images 202 of FIG. 2 further emphasize the faces of individuals (some of which also include bicycles).

[0073] For identification Figure 2 The digital image matching system receives an image mask 204 for selecting a similar digital image 202 of the query digital image 102. Specifically, the digital image matching system receives a selection of a portion of the query digital image 102, and the digital image matching system uses the selection to modify the deep neural network-based representation of the query digital image 102. As shown, the image mask 204 depicts the outline of the face of the person shown in the query digital image 102. Therefore, the digital image matching system uses the deep neural network-based model to generate a weighted deep neural network-based representation of the digital image 102. For example, in the deep neural network-based representation, the digital image matching system weights the portion of the query digital image 102 defined by the image mask 204 (e.g., features corresponding to the portion).

[0074] To illustrate, the digital image matching system modifies the deep neural network-based representation to take into account weighting the portion of the query digital image 102 indicated by the image mask 204. For example, the digital image matching system can utilize a modified spatial selectivity algorithm to generate a modified (e.g., weighted) deep neural network-based representation in the form of:

[0075]

[0076] Wherein m represents an image mask (eg, image mask 204).

[0077] like Figure 2 As shown, the digital image matching system may receive the image mask 204 as input by the user through the user interface 100. In practice, the digital image matching system may provide various tools within the user interface 100 through which the user may define the image mask 204. For example, although Figure 2 Image mask 204 is shown as an outline around the jumping woman in query digital image 102, but the digital image matching system may provide a tracking tool, a lasso tool, a bounding box tool, a brush tool, a coordinate positioning tool, a highlighter tool, or some other tool by which a user can define a portion of the query digital image 102 for image mask 204. Thus, in some embodiments, image mask 204 may have a shape similar to that of the jumping woman in query digital image 102. Figure 2 Different appearance shown in.

[0078] However, in some embodiments, the digital image matching system may automatically (e.g., without user input) determine m by a saliency detection method to generate a saliency map whose pixel values ​​correspond to region importance. Thus, the digital image matching system may intelligently determine whether to implement a weighted deep neural network-based representation to consider the image mask by analyzing the query digital image 102 according to the saliency detection method.

[0079] As mentioned above, and as Figure 2 As shown, similar digital images 202 and Figure 1 In fact, as a result of utilizing the modified deep neural network-based representation, whereby the digital image matching system weights the portion of the query digital image 102 indicated by the image mask 204, the digital image matching system identifies similar digital images 202 that emphasize features corresponding to the face shown within the query digital image 102. As shown, the similar digital images 202 each depict the face of at least one person, and for Figure 1 This is not the case with the similar digital images 104. In fact, due to the use of an unweighted deep neural network-based representation, one of the similar digital images 104 has a bicycle but no arbitrary person at all.

[0080] although Figure 2 Results using a particular deep neural network-based representation are shown (e.g., as a result of a spatially selective algorithm), but in some embodiments, the digital image matching system can generate a different deep neural network-based representation. For example, in the event that a user selects a different visual attribute option 106 (e.g., "synthesize" or "count"), the digital image matching system can generate a corresponding deep neural network-based representation. For illustration, in addition to or as an alternative to the above-described spatially selective algorithm, the digital image matching system can utilize an image synthesis algorithm and / or an object counting algorithm. Specific reference is made below to Figure 6 Provides additional details on the image synthesis algorithm and the object counting algorithm.

[0081] As described above, the digital image matching system may receive more than one query digital image, and in response to receiving multiple query digital images, may generate multiple query vector representations. Figure 3 A user interface 300 including more than one query digital images (eg, query digital images 302 and 308 ), similar digital images 304 , visual attribute options 306 , and image masks 310 is shown.

[0082] like Figure 3 As shown, the digital image matching system initially receives a first query digital image 308 (e.g., "layer 1") and subsequently receives a second query digital image 302 (e.g., "layer 2"). In addition, the digital image matching system can analyze each of the query digital images 308 and 302 to identify similar digital images 304 from the digital image database. To illustrate, similar to the above description of Figure 1-2 As discussed in detail above, the digital image matching system can utilize a deep neural network-based model to generate multiple query vector representations of query digital images 308 and 302 to serve as a basis for identifying similar digital images 304. In practice, the digital image matching system can identify similar digital images 304 as images within the database that share similar features of the multiple query vector representations used for query digital images 308 and 302.

[0083] More specifically, to generate a multiple query vector representation of query digital images 308 and 302, the digital image matching system generates a composite feature vector representing visual attributes of query digital image 308 and visual attributes of query digital image 308. To elaborate, the digital image matching system balances each of query digital images 308 and 302 so that no query digital image dominates (e.g., exceeds) any other query digital image. For example, in some embodiments, the digital image matching system performs a re-ranking technique by taking a plurality of top single-query search results (e.g., a first set of digital images similar to query digital image 308) and re-ranking the single-query search results using similarity scores indicating similarity to other single-query search results (e.g., a second set of digital images similar to query digital image 302).

[0084] More specifically, the digital image matching system can be based on the above Figure 1-2 Techniques and methods are described to generate a deep neural network based model and identify similar digital images for a query digital image 308. Similarly, a digital image matching system can generate a deep neural network based model and identify similar digital images for a query digital image 302. In these embodiments, the digital image matching system does not generate multiple query representations of the query digital images 308 and 302 together, but rather identifies similar images independently of each other.

[0085] On the other hand, in some embodiments, the digital image matching system performs a feature aggregation technique with respect to the above-described multiple query vector representation. To elaborate, the digital image matching system may aggregate the feature vectors of the query digital images rather than performing a separate search using the feature vectors of multiple query digital images (e.g., as re-ranking does). For example, the digital image matching system may aggregate or combine the feature vector associated with the query digital image 308 with the feature vector associated with the query digital image 302. Thus, the digital image matching system may generate a multiple query vector representation in the form of a composite feature vector that represents a synthesis of two feature vectors (e.g., the feature vectors of the query digital images 308 and 302).

[0086] In some embodiments, the digital image matching system averages the feature vectors of query digital images 308 and 302. However, in other embodiments, the digital image matching system utilizes different techniques for feature vectors that do not have similar value ranges or whose vector norms are substantially different (e.g., where feature vectors with larger values ​​or norms will dominate the search results, resulting in an imbalance in the contribution of the query digital image). Specifically, the digital image matching system generates a multiple query vector representation by normalizing each feature vector (e.g., a feature vector associated with query digital image 308 and a feature vector associated with query digital image 302) before averaging the feature vectors.

[0087] To illustrate, let y (1) ,y (2) ,...,y (N) represents the feature vectors of N query digital images. Therefore, the digital image matching system generates multiple query vector representations y by utilizing the feature aggregation technique given by multi :

[0088]

[0089] The first of these The average and second term of the L2-normalized feature vectors corresponding to multiple query digital images Denormalized average normalized feature vector. By utilizing the second term (e.g., the denormalization step), the digital image matching system makes the vector norm compatible with the norm of the digital images in the digital image database, as described in Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun, Optimized Product Quantization for Approximate Nearest Neighbor Search, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2946-2953 (2013), the entire contents of which are incorporated herein by reference.

[0090] As described above, the digital image matching system can apply aggregation techniques to any of the deep neural network based representations described herein. In practice, the digital image matching system can modify the deep neural network based representations for multi-query scenarios as well as single-query scenarios by utilizing one or more of a spatial selectivity algorithm, an image synthesis algorithm, and / or an object counting algorithm. To merge features of the digital image, in some embodiments, the digital image matching system can average the normalized multi-query features and then perform a non-normalization step.

[0091] In other embodiments, the digital image matching system can generate similar digital images for the query digital image 308 independently of the query digital image 302. Likewise, the digital image matching system can generate a second similar digital image set about the query digital image. In addition, the digital image matching system can compare the relative rankings of digital images similar to the query digital image 308 and digital images similar to the query digital image 302. The digital image matching system can also determine the similarity scores of the multiple query vector representations of the query digital images 308 and 302 based on the comparison of relative rankings. Therefore, based on determining the similarity score of the query digital image 302 after the query digital image 308 is completed, the digital image matching system can rearrange the search results based on the comparison of relative rankings. In fact, the digital image matching system can rank those digital images associated with the query digital image 308 in the search results, and those digital images have higher similarity scores than the digital images associated with the query digital image 302.

[0092] like Figure 3 As shown, the digital image matching system can receive an image mask 310 corresponding to the query digital image 308. As shown, the image mask 310 indicates a portion of the query digital image 308 depicting a tree. Therefore, as described above, the digital image matching system generates a modified deep neural network-based representation that takes into account the weighting of the portion of the query digital image 308 indicated by the image mask 310.

[0093] Thus, as shown, the digital image matching system identifies similar digital images 304 that each contain a tree. In addition, because the digital image matching system receives not only the query digital image 308 and the corresponding image mask 310, but also the query digital image 302 depicting a plurality of animals, the digital image matching system further identifies some similar digital images 304 that include animals.

[0094] As described above, the digital image matching system may receive more than one image mask. In fact, in addition to the image mask corresponding to the query digital image 302, the digital image matching system may also receive the image mask 310. Figure 4 As shown, the digital image matching system may receive the same query digital images 308 and 302 except for the image masks 310 and 404 .

[0095] Figure 4 User interface 300 is shown including query digital image 302, query digital image 308, image mask 310, image mask 404, similar digital images 402, weighting options 406 and 408, invert mask option 410, and clear mask option 412. Figure 3 As discussed above, the digital image matching system receives a query digital image 308 along with an accompanying image mask 310 and the query digital image 302. In addition, the digital image matching system receives an image mask 404 that indicates a portion of the query digital image 302 to emphasize when generating a multiple query vector representation. As shown, the digital image matching system receives the image mask 404 that depicts the outline of a dog within the query digital image 302.

[0096] Based at least in part on receiving image mask 404, the digital image matching system identifies similar digital images 402 that depict dogs, particularly dogs that appear similar to the dog of image mask 404. For example, the digital image matching system applies the above-described techniques to utilize a deep neural network-based model to generate a modified deep neural network-based representation that aggregates features of query digital image 308 with features of query digital image 302 to generate a composite feature vector, and further applies weights to image mask 310 and image mask 404. By analyzing query digital images 308 and 302 in this manner, the digital image matching system can search the digital image database to identify similar digital images 402 that depict a tree (e.g., as indicated by image mask 310) and at least one dog (e.g., as indicated by image mask 404).

[0097] like Figure 4 As further shown in FIG. 3 , user interface 300 includes weight options 406 and 408. Specifically, user interface 300 shows each weight option 406 and 408 having a weight of 50, i.e., each of query digital images 308 and 302 is uniformly weighted. However, the digital image matching system may receive user input for modifying the weight associated with query digital image 308 and user input for modifying the weight associated with query digital image 302. For example, the digital image matching system may receive user input related to weight option 406 to adjust the weight given to query digital image 308. Similarly, the digital image matching system may receive user input related to weight option 408 to adjust the weight given to query digital image 302.

[0098] Based on receiving user interaction / input related to weight options 406 and / or 408, the digital image matching system modifies the deep neural network-based representation (e.g., the multiple query vector representation). Specifically, the digital image matching system modifies the deep neural network-based representation to adjust the influence of the query digital images 308 and 302 as part of identifying similar digital images. For example, in response to receiving user input to increase the weight option 406, the digital image matching system weights the features of the query digital image 308 accordingly as part of generating a composite feature vector. Therefore, as a result of the increased weight, the digital image matching system generates a multiple query vector representation (e.g., a composite feature vector) that is more reflective of the features of the query digital image 308 - that is, leaning toward the features of the query digital image 308.

[0099] Also like Figure 4 As shown, user interface 300 includes an invert mask option 410. In response to receiving a user interaction (e.g., clicking, tapping, touching, etc.) regarding invert mask option 410, the digital image matching system inverts the image mask. For illustration, the digital image matching system inverts the image mask corresponding to the currently selected query digital image. For example, Figure 4 As shown, the query digital image 302 is currently selected. Thus, in response to receiving a user interaction with the invert mask option 410, the digital image matching system inverts the image mask 404 of the query digital image 302.

[0100] To invert the image mask 404, the digital image matching system considers all portions outside the image mask 404 as masked portions of the query digital image 302. To elaborate, typically the digital image matching system emphasizes (e.g., weights) those portions of the query digital image 302 indicated within the image mask 404 to search for and identify similar digital images 402. However, in response to receiving a user selection of the invert mask option 410, the digital image matching system performs the opposite operation by weighting those portions of the query digital image 302 that are outside the image mask 404 (effectively masking those portions of the query digital image 302). Thus, the digital image matching system can invert the image mask 404 to emphasize portions of the query digital image 302 that are outside the location where the user defined the image mask 404.

[0101] like Figure 4 As further shown in FIG. 4 , the user interface 300 includes a clear mask option 412. In response to receiving a user selection or other input related to the clear mask option 412, the digital image matching system clears the image mask corresponding to the selected query digital image. Figure 4As shown, the query digital image 302 is currently selected. Therefore, in response to receiving the user's selection of the clear mask option 412 , the digital image matching system clears (eg, removes, deletes, etc.) the image mask 404 from the query digital image 302 .

[0102] When the image mask is cleared 404, the digital image matching system can receive the user's selection of a new image mask and can then identify new similar digital images. Figure 5 A user interface 300 is shown, which includes Figure 4 302, but with a new image mask 504 over the query digital image 302. Based on receiving the image mask 504 indicating a portion of the query digital image 302 depicting an elephant, the digital image matching system utilizes a deep neural network-based model including a deep neural network-based representation that considers each query digital image and the corresponding image mask to search for and identify similar digital images 502 that depict an elephant and a tree together. In fact, as described above, the digital image matching system can identify similar digital images that more closely resemble the portion of the query digital image depicted within the image mask.

[0103] Although Figure 1-5 The user selection of the visual attribute option "similarity" is shown, but in some embodiments, in addition to or as an alternative to "similarity", the user selects a different visual attribute option. Therefore, in addition to or as an alternative to the selective similarity algorithm, the digital image matching system can utilize a different algorithm to generate a deep neural network based representation. For example, Figure 6 As shown, in some embodiments, the digital image matching system receives a user selection of a visual attribute option of "synthetic" and / or "count" visual attribute option 606. Based on receiving the user selection of the visual attribute option 606 corresponding to the object counting analysis (e.g., the "count" option), the digital image matching system utilizes the object counting algorithm.

[0104] As described above, the digital image matching system utilizes an object counting algorithm in response to receiving a user selection of the "Count" visual attribute option 606. In practice, Figure 6A user interface 600 is shown, which includes a query digital image 602, similar digital images 604, and a visual attribute option 606, where "count" is selected. Based on determining that "count" is selected, the digital image matching system uses a sense number technique to count objects within the query digital image 602. In some embodiments, the digital image matching system uses a sense number network to generate features that strongly correspond to object counts as part of an object counting algorithm for generating a deep neural network-based representation. For example, the digital image matching system implements the sense number network described in Jianming Zhang, Shugao Ma, Mehrnoosh Sameki, Stan Sclaroff, Margrit Betke, Zhe Lin, Xiaohui Shen, Brian Price, and Radomir Mech, Salient Object Subitizing, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4045-54 (2015), the entire contents of which are incorporated herein by reference. As described above, the digital image matching system employs network activations (eg, 128-dimensional feature vectors) before the final classification layer as descriptors for identifying similar digital images 604 for similarity search.

[0105] By utilizing an object counting algorithm, the digital image matching system identifies similar digital images 604 based on the query digital image 602. Specifically, the digital image matching system determines similarity scores for multiple digital images within the digital image database based on the object counts. Based on the similarity scores, the digital image matching system ranks the multiple digital images to identify similar digital images 604 that depict the same (or similar) number of objects as the query digital image 602.

[0106] Although not in Figure 6 , but the digital image matching system may additionally or alternatively receive a user selection of a “synthetic” visual attribute option 606. Based on receiving the user selection of the “synthetic” visual attribute option 606, the digital image matching system may utilize an image synthesis algorithm to generate a deep neural network-based representation of the query digital image 602.

[0107] In detail, as part of the image synthesis algorithm, the digital image matching system spatially transforms the feature map of the above baseline The global aggregation may sometimes discard spatial structures with the information of the query digital image and the feature map. Therefore, searching for visually similar digital images using only the global aggregation features may produce similar digital images that are similar in terms of semantic content, without placing too much emphasis on the arrangement of the objects and / or background of the query digital image 602. For example, two different query digital images of the same object but from different vantage points may result in similar global aggregation features.

[0108] Therefore, in order to achieve more accurate visual similarity search based on image composition (such as arrangement, layout or positioning), digital image matching systems implement image composition algorithms to spatially aggregate feature maps into highly and width To generate a deep neural network-based representation. As mentioned above, the digital image matching system uses a deep neural network-based model to generate feature maps for synthetic similarity search.

[0109] In order to achieve a balance between spatial structure preservation and spatial constraints, the digital image matching system can use h comp and w comp For example, in some embodiments, the digital image matching system sets h comp and w comp Each of h is set to 3. In other embodiments, the digital image matching system sets h comp and w comp Set to some other value (e.g. 5, 10, etc.).

[0110] In addition, for a feature map of a query digital image (e.g., query digital image 602) of arbitrary spatial size, the digital image matching system utilizes an adaptive spatial pooling technique to uniformly pool the feature map y into a 3×3 vector. For example, in some embodiments, the digital image matching system implements the adaptive social pooling described in Kaiming He, Xiangyu Zhang, Shaoquing Ren, and Jian Sun, Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, European Conference on Computer Vision, Springer, 346-61 (2014), the entire contents of which are incorporated herein by reference.

[0111] As described above, the digital image matching system utilizes a similarity search function F(·) to identify similar digital images (e.g., similar digital image 604). For synthetic similarity based on an image synthesis algorithm, the digital image matching system utilizes a position similarity search function where i and j are position indices, as described above. Position similarity search function Each instance of processes the visual similarity of its corresponding position within the query digital image 602.

[0112] Compared with a single similarity database, the digital image matching system utilizes multiple similarity databases for image synthesis analysis, each similarity database storing a specific position (e.g., h) of a digital image corresponding to a digital image in the digital image database. comp × comp In addition, in order to determine the synthetic similarity score S comp , the digital image matching system transforms the feature map y cp h comp × comp The feature vectors at different spatial locations within the grid are passed to F(·). The digital image matching system aggregates the similarity scores of the feature vectors according to the following formula:

[0113]

[0114] Similar to the discussion above regarding spatial selectivity algorithms, the digital image matching system can further consider image masks indicated within the query digital image as part of the image composition analysis. In fact, the digital image matching system can modify the image composition algorithm to enhance spatial selectivity and give importance to the y-images detected via automatic saliency. cp For example, given an image mask weight m, the digital image matching system aggregates the similarity scores of the feature vectors of the digital images in the digital image database according to the following formula:

[0115]

[0116] In some embodiments, the digital image matching system receives a user selection of more than one visual attribute option 606. In these embodiments, the digital image matching system generates a deep neural network-based representation that takes into account each selected visual attribute option 606. For example, the digital image matching system can generate a combined or composite deep neural network-based representation in the form of a linear combination of feature vectors corresponding to each selected visual attribute option 606. In some embodiments, the digital image matching system adds together the feature vectors corresponding to each selected visual attribute option 606 to generate the combined deep neural network-based representation.

[0117] Additionally, the digital image matching system may aggregate multiple visual similarities by determining a weighted sum of similarity scores associated with the spatial selectivity algorithm, the image synthesis algorithm, and / or the object counting algorithm. In practice, the digital image matching system determines the weighted sum of similarity scores as:

[0118] S agg =w sem S sem +w comp S comp +w subi S subi

[0119] Where S agg is the aggregate similarity score, S sem is a semantic similarity score (e.g. related to spatial selectivity algorithms), S comp is the image synthesis similarity score, S subi is the object count similarity score (i.e., sense similarity score), w sem is the weight associated with the semantic similarity score, w comp is the weight associated with the image synthesis similarity score, and w subi is the weight associated with the object count / sense similarity score.

[0120] In some embodiments, the digital image matching system utilizes sem 、w comp and w subi A preset value (e.g., user-defined or system-defined via user interface 600) of Figure 6 6, but in addition to (or as an alternative to) the check boxes for visual attribute options 606, user interface 600 may also include attribute weight options through which the user can adjust the weight of each individual visual attribute. sem 、w comp , and w subi .

[0121] In some embodiments, the digital image matching system applies a weight of 0 to any visual attribute corresponding to an unchecked visual attribute option 606. Figure 6 As explained in the figure, the digital image matching system is set up w sem = 0 and w comp =0, because the only visual attribute option 606 checked is "Count".

[0122] As further mentioned, the digital image matching system can provide similar digital images 604 based on similarity scores related to the query digital image 602. Specifically, the digital image matching system can rank the digital images within the digital image database based on relevance to the query digital image 602. For example, the digital image matching system can rank the digital images within the digital image database based on the similarity scores (e.g., S ) compared to the query digital image 602. sem , S comp and / or S subi ) may rank similar digital images, and may provide similar digital images 604 for display within the user interface 600 based on their relative rankings.

[0123] although Figure 1-6 Specific layouts and appearances of various user interfaces (e.g., user interfaces 100, 300, and 600) are shown, but in some embodiments, the digital image matching system may provide user interfaces having different appearances and including Figure 1-6 600. For example, user interface 600 may have a different arrangement, wherein visual attributes option 606 is located at a different location within user interface 600, and wherein invert mask option and clear mask option are, for example, in a drop-down menu or tool bar associated with creating an image mask. Additional or alternative arrangements are also possible.

[0124] As described above, the digital image matching system utilizes a deep neural network based model to generate a deep neural network based representation for identifying similar digital images. In some embodiments, the digital image matching system further trains the deep neural network based model. In practice, Figure 7 Training a deep neural network based model according to one or more embodiments is shown.

[0125] like Figure 7 As shown, the digital image matching system trains a deep neural network-based model 704 based on training data including training digital images and image features. To train the deep neural network-based model 704, the digital image matching system accesses a digital image database 714 to identify a training digital image 702 to use as an input to the deep neural network-based model 704. Based on the training digital image 702, the digital image matching system generates predicted image features 706. More specifically, as described above, the deep neural network-based model 704 generates a predicted deep neural network-based representation corresponding to the training digital image 702.

[0126] In addition, the digital image matching system compares the predicted image features 706 to the actual image features 710. In detail, the digital image matching system compares the known depth features of the training digital image 702 to the predicted depth features of the same image. To compare the predicted image features 706 to the actual image features 710, the digital image matching system applies a loss function 708. In practice, by applying the loss function, the digital image matching system determines an error or loss associated with the predicted image features. In addition, the digital image matching system performs an error reduction technique 712 to reduce the determined error or loss by modifying the parameters of the deep neural network based model 704.

[0127] As shown, the digital image matching system repeats the training process by accessing a second training digital image from the digital image database 714 and generating a corresponding set of predicted features to compare with the actual features. Figure 7 The training process shown in the figure continuously reduces the loss of the current prediction by adjusting the network parameters, and the digital image matching system improves the accuracy of the deep neural network based model 704 to a point where the deep neural network based model 704 generates predicted image features that match (or are within an error threshold) the corresponding actual image features when the training process is completed.

[0128] See now Figure 8 , will provide additional details about the components and capabilities of the digital image matching system. Specifically, Figure 8 An example schematic diagram of a digital image matching system 802 within (eg, as a part of) an image management system 801 on an example computing device 800 (eg, a user client device and / or a server device) is shown. Figure 8 As shown, the digital image matching system 802 may include features of a query digital image manager 804 , a deep neural network based model manager 806 , a similarity determination manager 808 , a digital image provider 810 , and a storage manager 812 .

[0129] As described above, the digital image matching system 802 includes a query digital image manager 804. Specifically, the query digital image manager 804 can communicate with a user client device to receive and / or detect a query digital image, along with any user selections of visual attributes, weights, image masks, etc. For example, the query digital image manager 804 can receive a query digital image along with an image mask, and can communicate with a deep neural network-based model manager 806 to provide an indication for using a deep neural network-based representation that corresponds to the user's selection of visual attributes and weights the image mask of the query digital image accordingly.

[0130] In addition, the digital image matching system 802 includes a deep neural network-based model manager 806. Specifically, the deep neural network-based model manager 806 can manage, analyze, or otherwise process the query digital image to generate a deep neural network-based representation of the digital image. For example, the deep neural network-based model manager 806 can utilize a deep neural network-based model to generate a set of deep features and further utilize one of the above algorithms (e.g., a selective similarity algorithm, an image synthesis algorithm, or an object counting algorithm) to generate a deep neural network-based representation. In addition, the deep neural network-based model manager 806 can communicate with the similarity determination manager 808 to provide the similarity determination manager with information related to image features.

[0131] As described above, the digital image matching system 802 includes a similarity determination manager 808. Specifically, the similarity determination manager 808 can utilize a similarity search algorithm to search the digital image database and identify visually similar digital images for the query digital image. For example, the similarity determination manager 808 can communicate with the storage manager 812 to determine the similarity scores of all (or a subset) of the digital images within the digital image database 814. In addition, the similarity determination manager 808 can rank the digital images within the digital image database 814 based on their respective similarity scores associated with a particular query digital image. In some embodiments, the storage manager 812 can store a single digital image database 814 containing deep features of candidate digital images, while in other embodiments, the storage manager 812 can store multiple databases, including a database of multi-query feature information for matching within a multi-query scenario.

[0132] As shown, the digital image matching system 802 also includes a digital image provider 810. Specifically, the digital image provider 810 can communicate with the similarity determination manager 808 to provide, present, display or otherwise send similar digital images to the user client device. For example, the digital image provider 810 can provide a plurality of top-ranked similar digital images (e.g., digital images that exceed a similarity threshold). In these or other embodiments, the digital image provider 810 provides similar digital images for the user client device to present in the order of their respective similarity scores (e.g., the highest score first).

[0133] In one or more embodiments, each component of the digital image matching system 802 communicates with each other using any suitable communication technology. In addition, the components of the digital image matching system 802 can communicate with one or more other devices, including the following combined Fig. 9 One or more user client devices described herein. It will be appreciated that although Figure 8Components of the digital image matching system 802 are shown as separate, but any of the subcomponents can be combined into fewer components, such as into a single component, or can be divided into more components for a particular implementation. In addition, although the digital image matching system 802 is described in conjunction with Figure 8 , but at least some of the components used to perform operations in conjunction with the digital image matching system 802 described herein may be implemented on other devices within the environment.

[0134] The components of the digital image matching system 802 may include software, hardware, or both. For example, the components of the digital image matching system 802 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., computing device 800). When executed by one or more processors, the computer-executable instructions of the digital image matching system 802 may enable the computing device 800 to perform the block matching method described herein. Alternatively, the components of the digital image matching system 802 may include hardware, such as a dedicated processing device for performing a specific function or group of functions. Additionally or alternatively, the components of the digital image matching system 802 may include a combination of computer-executable instructions and hardware.

[0135] In addition, the components of the digital image matching system 802 that perform the functions described herein can be implemented, for example, as part of a stand-alone application, as a module of an application, as a plug-in for an application including a content management application, as a library function or function that can be called by other applications, and / or as a cloud computing model. Therefore, the components of the digital image matching system 802 can be implemented as part of a stand-alone application on a personal computing device or a mobile device. Alternatively or additionally, the components of the digital image matching system 802 can be implemented in any application that allows the creation and delivery of marketing content to users, including but not limited to applications in ADOBE CREATIVE CLOUD, such as ADOBE PHOTOSHOP, ADOBE STOCK, ADOBE LIGHTROOM, ADOBE EXPERIENCE MANAGER and ADOBE PREMIERE, "ADOBE", "PHOTOSHOP", "STOCK", "LIGHTROOM", "EXPERIENCE MANAGER" and "PREMIERE" are registered trademarks of Adobe Systems Incorporated in the United States and / or other countries.

[0136] Fig. 9A schematic diagram of one embodiment of an exemplary environment in which the digital image matching system 802 operates is shown. In some embodiments, the exemplary environment includes a network 902, a server 904, and a user client device 906 including an image management application 908. Although Fig. 9 Particular arrangements of components of the environment are shown, but the environment may have additional or alternative arrangements.

[0137] like Fig. 9 As shown, the environment may include a network 902. Specifically, the network 902 may connect a server 904 and a user client device 906. Thus, the network 902 may facilitate communication between the server 904 and the user client device 906 via appropriate network protocols. For example, the network 902 may refer to a local network (e.g., a local area network or "LAN") or a wide area network ("WAN"), or may refer to different communication protocols through which two computing devices may communicate.

[0138] Also like Fig. 9 As shown in , the environment may include a server 904. The server 904 may refer to at least one computing device (e.g., a virtual reality or augmented reality device) that can manage a deep neural network-based model to analyze digital images and identify similar digital images. Specifically, the server 904 may communicate with an image management application 908 on (e.g., housed therein or installed thereon) a user client device 906 via the network 902. Thus, the server 904 may communicate with the image management application 908 to enable a user to interact with a user interface provided by the image management application 908 to view, edit, select, or otherwise interact with query digital images and other user interface options such as visual attribute options and image masks.

[0139] In some embodiments, the digital image matching system 802 may be implemented as part of the image management system 801 on the user client device 906, and may therefore communicate directly with the user client device 906. Thus, the digital image matching system 802 may enable a user to analyze a query digital image and identify similar digital images as part of a native application installed locally on the user client device 906, as hardware, software, or both.

[0140] although Fig. 9 A specific arrangement of an environment including a server 904, a network 902, and a user client device 906 is shown, but various additional or alternative arrangements are possible. Fig. 9 A single user client device 906 is shown communicating with the network 902 and the server 904 , but in some embodiments, multiple user client devices may bypass the network 902 and communicate directly with the server 904 .

[0141] For illustration, the user client device 906 can communicate with the server 904 to display video content provided by the server 904. For example, the user client device 906 can display a user interface provided by the server 904 that includes a query digital image, similar digital images, and / or other user interface options. The server 904 utilizes the digital image matching system 802 to analyze the query digital image to identify similar digital images in accordance with the present disclosure. The server 904 can communicate with the user client device 906 to draw or otherwise display any changes or modifications made to the user interface.

[0142] Figure 1-9 , corresponding text and examples provide a number of different systems, methods, and non-transitory computer-readable media for identifying similar digital images based on various visual attributes using a deep neural network-based model. In addition to the foregoing, embodiments may also be described in terms of flowcharts that include actions for achieving a particular result. For example Fig.10 A flow diagram illustrating an example sequence of actions in accordance with one or more embodiments is shown.

[0143] Although Fig.10 Actions according to one embodiment are shown, but alternative embodiments may omit, add, re-rank and / or modify Fig.10 Any action shown in . Fig.10 The actions of may be performed as part of a method. Alternatively, a non-transitory computer readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform Fig.10 In another embodiment, the system may execute Fig.10 Alternatively, the actions described herein may be repeated or performed in parallel with each other, or with different instances of the same or other similar actions.

[0144] Fig.10 An example series of actions 1000 for identifying similar digital images based on various visual attributes using a deep neural network-based model is shown. Specifically, the series of actions 1000 includes an action 1002 of receiving a query digital image and a user selection. For example, action 1002 may involve receiving a user selection of a query digital image from a user client device and at least one of spatial selectivity, image synthesis, or object counting for identifying similar digital images. As described above, the digital image matching system may receive a query digital image input by a user through a user interface on a user client device.

[0145] Additionally, the series of actions 1000 may include an action of generating a deep feature set 1004. Specifically, action 1004 may involve utilizing a deep neural network based model to generate a deep feature set for the query digital image.

[0146] As shown, a series of actions 1000 may also include an action 1006 of generating a deep neural network-based representation. Specifically, action 1006 may involve generating a deep neural network-based representation of the query digital image based on the deep feature set of the query digital image and according to a user selection by utilizing one or more of a spatial selective algorithm, an image synthesis algorithm, or an object counting algorithm. Action 1006 may also involve modifying the deep feature set by utilizing a deep neural network-based representation including at least one of a spatial selective algorithm, an image synthesis algorithm, or an object counting algorithm.

[0147] like Fig.10 As further shown in , the series of actions 1000 may include an action of identifying similar digital images 1008. Specifically, action 1008 may involve identifying similar digital images for the query digital image from a digital image database based on a deep neural network-based representation of the query digital image.

[0148] Although not in Fig.10 , but the series of actions 1000 may also include the action of training a deep neural network-based model to generate deep features for a digital image. The series of actions 1000 may also further include the action of receiving an image mask from a user client device, the image mask indicating a portion of the query digital image to emphasize when identifying similar digital images. In addition, the series of actions 1000 may include the action of generating a deep neural network-based representation based on weighting the portion of the query digital image indicated by the image mask in response to receiving the image mask. Weighting the portion of the query digital image indicated by the image mask may include applying weights to features of the feature set corresponding to the portion of the query digital image indicated by the image mask.

[0149] The series of actions 1000 may include the actions of determining similarity scores of a plurality of digital images from a digital image database and ranking the plurality of digital images based on the determined similarity scores. In addition, the series of actions may include the following actions: receiving a selection of a second query digital image from a user client device; generating a second deep feature set for the second query digital image using a deep neural network-based model; generating a composite multi-query vector representation representing the query digital image and the second query digital image based on the second deep feature set of the second query digital image; and identifying similar digital images from the digital image database for the multi-query vector representation.

[0150] In addition, the series of actions 1000 may include an action of receiving a second user selection of at least one of spatial selection, image synthesis, or object counting with respect to a second query digital image, wherein the instructions cause the computer device to generate a multiple query vector representation based on the second user selection by utilizing at least one of a spatial selectivity algorithm, an image synthesis algorithm, or an object counting algorithm. The series of actions 1000 may also include an action of receiving an image mask from a user client device and a second image mask, the image mask indicating a portion of the query digital image to emphasize when identifying similar digital images, the second image mask indicating a portion of the second query digital image to emphasize when identifying similar digital images.

[0151] Additionally, the series of actions 1000 may include an action of generating a multiple query vector representation in response to receiving the image mask and the second image mask to emphasize the portion of the query digital image indicated by the image mask and the portion of the second query digital image indicated by the second image mask. The series of actions 1000 may further include an action of comparing the depth features of each of the multiple digital images with the depth features of the multiple query vector representation.

[0152] In addition, the series of actions 1000 may include an action of determining similarity scores of multiple digital images within a digital image database based on a deep neural network-based representation of the query digital image. Determining the similarity score may include comparing the deep features of each of the multiple digital images with the deep features of the query digital image. Therefore, the series of actions 1000 may also include the following actions: receiving a selection of a second query digital image from a user client device; generating a second set of deep features for the second query digital image using a deep neural network-based model; generating a composite multi-query vector representation representing the query digital image and the second query digital image based on the second set of deep features of the second query digital image; determining similarity scores of multiple digital images within a multi-query digital image database based on the deep neural network-based representation of the second query digital image; and identifying similar digital images from the digital image database for the multi-query vector representation.

[0153] In addition, the series of actions 1000 may include the following actions: receiving from a user client device: an image mask indicating a portion of a query digital image to emphasize when identifying similar digital images; a second image mask indicating a portion of a second query digital image to further emphasize when identifying similar digital images; a first weight for application to the query digital image; a second weight for application to the second query digital image; and modifying the multi-query vector representation to apply the first weight to the portion of the query digital image indicated by the image mask, and apply the second weight to the portion of the second query digital image indicated by the second image mask.

[0154] As mentioned above, digital image matching systems can use models based on deep neural networks to identify similar digital images. In fact, Fig.11 The steps for identifying similar digital images to a query digital image according to one or more embodiments of the digital image matching system described herein are shown. In addition to the related methods and techniques described elsewhere in this disclosure, the steps for determining the depth of feature points within the input video may include: Fig.11 The following description.

[0155] like Fig.11 As shown, the steps for determining the depth of feature points within the input video may include actions 1102-1116. In practice, actions 1102-1116 may each indicate the behavior of utilizing a model based on a deep neural network. Specifically, the digital image matching system may perform action 1102 to determine the visual attributes of the query digital image. In practice, action 1102 may include determining visual attributes such as semantic content, spatial selectivity, image synthesis, object count, color, etc. based on receiving user input as described herein.

[0156] As shown, the digital image matching system may also perform action 1104 to determine whether the user has created an image mask. In practice, the digital image matching system may determine whether the digital image matching system receives an image mask from a user client device.

[0157] Based on determining that the user created an image mask, the digital image matching system performs action 1106 to identify a portion of the query digital image indicated by the image mask. Additionally, the digital image matching system performs action 1108 to generate and / or modify a deep neural network-based representation (or use a modified deep neural network-based representation) to weight the portion of the query digital image indicated by the image mask.

[0158] In addition, the digital image matching system performs action 1112 to determine similarity scores of the plurality of digital images. For example, the digital image matching system implements a similarity search function (e.g., F(·) as described above) to determine similarity scores of the plurality of digital images within the digital image database. The digital image matching system also performs action 1114 to rank the digital images according to their respective similarity scores, and performs action 1116 to identify the digital image with the highest similarity score as a similar digital image.

[0159] like Fig.12As shown, the digital image matching system may also perform steps for identifying similar digital images for the composite feature vector. In practice, the digital image matching system may perform actions 1202-1218 as part of (or separate from) utilizing a model based on a deep neural network. For example, the digital image matching system may perform action 1202 to determine visual attributes of a query digital image and a second query digital image. In practice, in response to receiving a plurality of query digital images, the digital image matching system may determine visual attributes of each query digital image.

[0160] The digital image matching system may also perform action 1204 to generate a composite feature vector. For example, the digital image matching system may aggregate features associated with each query digital image to generate a composite feature vector that represents a synthesis of features of each query digital image according to the present disclosure. Fig.11 As discussed above, the digital image matching system may further perform action 1206 to determine whether the user has created an image mask. In practice, the digital image matching system may receive more than one image mask and may receive more than one image mask for the query digital image.

[0161] As described above, in response to determining that the user did create at least one image mask, the digital image matching system may perform action 1208 to identify the portion of the query digital image indicated by the image mask, and further perform action 1210 to generate and / or modify a deep neural network-based representation to weight the indicated portion of the query digital image.

[0162] In addition, the digital image matching system may perform action 1214 to determine similarity scores of the plurality of digital images. For example, the digital image matching system may implement a similarity search function as described above. In addition, the digital image matching system may perform action 1216 to rank the plurality of digital images according to their respective similarity scores, and may perform action 1218 to identify those digital images with the highest similarity scores as similar digital images. In some embodiments, the digital image matching system may identify those digital images whose similarity scores satisfy a similarity threshold as similar digital images.

[0163] Embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Specifically, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein.

[0164] Computer-readable media can be any available media that can be accessed by a general or special purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct computer-readable media: a non-transitory computer-readable storage medium (device) and a transmission medium.

[0165] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code components in the form of computer-executable instructions or data structures and can be accessed by a general or special purpose computer.

[0166] A "network" is defined as one or more data links that support the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer correctly views the connection as a transmission medium. Transmission media may include networks and / or data links, which may be used to carry desired program code components in the form of computer executable instructions or data structures and may be accessed by general or special-purpose computers. Combinations of the above should also be included within the scope of computer-readable media.

[0167] In addition, upon reaching various computer system components, program code components in the form of computer executable instructions or data structures can be automatically transferred from the transmission medium to the non-transient computer readable storage medium (device) (or vice versa). For example, computer executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., "NIC") and then ultimately transferred to the computer system RAM and / or less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that non-transient computer readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0168] Computer executable instructions include, for example, instructions and data, which, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or function group. In some embodiments, computer executable instructions are executed on a general-purpose computer to turn a general-purpose computer into a special-purpose computer that implements an element of the present disclosure. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter is described in a language dedicated to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. On the contrary, the features and actions described are disclosed as example forms for implementing the claims.

[0169] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be implemented in a distributed system environment, where local and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or by a combination of hardwired and wireless data links) all perform tasks. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0170] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide universal and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be rapidly provisioned via virtualization and published with low management effort or service provider interaction, and then scaled accordingly.

[0171] The cloud computing model may consist of various features, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model may also expose various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this specification and claims, a "cloud computing environment" is an environment that employs cloud computing.

[0172] Fig.13 An example computing device 1300 (e.g., computing device 800, client device 906, and / or server 904) is shown in block diagram form, which may be configured to perform one or more of the processes described above. It will be appreciated that the digital image matching system may include an implementation of computing device 1300. Fig.13 As shown, the computing device may include a processor 1302, a memory 1304, a storage device 1306, an I / O interface 1308, and a communication interface 1310. In addition, the computing device 1300 may include an input device, such as a touch screen, a mouse, a keyboard, etc. In some embodiments, the computing device 1300 may include a processor 1302, a memory 1304, a storage device 1306, an I / O interface 1308, and a communication interface 1310. In addition, the computing device 1300 may include an input device, such as a touch screen, a mouse, a keyboard, etc. Fig.13 Fewer or more components may be shown. Fig.13 The components of the device 1300 shown in FIG. 1 will now be described in more detail.

[0173] In certain embodiments, processor 1302 includes hardware for executing instructions, such as instructions that constitute a computer program. As an example and not by way of limitation, to execute instructions, processor 1302 may retrieve (or fetch) instructions from internal registers, internal cache, memory 1304, or storage device 1306 and decode and execute them.

[0174] The computing device 1300 includes a memory 1304 coupled to the processor 1302. The memory 1304 may be used to store data, metadata, and programs for execution by the processor. The memory 1304 may include one or more of volatile and non-volatile memories, such as random access memory ("RAM"), read-only memory ("ROM"), solid state disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. The memory 1304 may be internal or distributed memory.

[0175] Computing device 1300 includes storage device 1306, which includes memory for storing data or instructions. By way of example, but not limitation, storage device 1306 may include the non-transitory storage media described above. Storage device 1306 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0176] The computing device 1300 also includes one or more input or output ("I / O") devices / interfaces 1308, which are provided to allow a user to provide input (e.g., user strokes) to the computing device 1300, receive output from the computing device 1300, and otherwise transfer data to and from the computing device 1300. These I / O devices / interfaces 1308 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of these I / O devices / interfaces 1308. The touch screen may be activated with a writing device or a finger.

[0177] I / O devices / interfaces 1308 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, devices / interfaces 1308 are configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or may be used as any other graphical content for a particular implementation.

[0178] The computing device 1300 may also include a communication interface 1310. The communication interface 1310 may include hardware, software, or both. The communication interface 1310 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices 800 or one or more networks. As an example and not by way of limitation, the communication interface 1310 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as WI-FI. The computing device 1300 may also include a bus 1312. The bus 1312 may include hardware, software, or both that couple the components of the computing device 1300 to each other.

[0179] In the foregoing description, the present invention has been described with reference to specific example embodiments of the present invention. Various embodiments and aspects of the present invention are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and the accompanying drawings are illustrative of the present invention and should not be construed as limiting the present invention. Many specific details are described to provide a thorough understanding of the various embodiments of the present invention.

[0180] Without departing from the spirit or essential features of the present invention, the present invention may be implemented in other specific forms. The described embodiments should be considered in all respects to be illustrative only and not restrictive. For example, the method described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in different orders. In addition, the steps / actions described herein may be performed repeatedly or in parallel with each other, or in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the present invention is represented by the appended claims rather than the preceding description. All changes within the meaning and equivalent scope of the claims are encompassed within their scope.

Claims

1. A non-transitory computer-readable medium for matching digital images based on visual similarity in a digital media environment for searching digital images, the non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause a computer device to: receiving from a user client device a user selection of a query digital image and at least one of spatial selectivity, image composition, or object count for identifying similar digital images; Generating a deep feature set for the query digital image using a deep neural network-based model; generating a deep neural network-based representation of the query digital image based on the set of deep features of the query digital image and in accordance with the user selection by utilizing one or more of a spatially selective algorithm, an image synthesis algorithm, or an object counting algorithm; as well as Based on the deep neural network based representation of the query digital image, similar digital images to the query digital image are identified from a digital image database.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to train the deep neural network-based model to generate deep features for digital images.

3. The non-transitory computer-readable medium of claim 1 , wherein the instructions cause the computer device to generate the deep neural network-based representation by modifying the set of deep features using a deep neural network-based representation comprising at least one of the spatial selectivity algorithm, the image synthesis algorithm, or the object counting algorithm.

4. The non-transitory computer-readable medium of claim 3, further comprising instructions that, when executed by the processor, cause the computer device to receive an image mask from the user client device, the image mask indicating a portion of the query digital image to be emphasized when identifying similar digital images.

5. The non-transitory computer-readable medium of claim 4, further comprising instructions that, when executed by the processor, cause the computer device to generate the deep neural network-based representation based on weighting the portion of the query digital image indicated by the image mask in response to receiving the image mask.

6. The non-transitory computer-readable medium of claim 5, wherein weighting the portion of the query digital image indicated by the image mask comprises applying weights to features of the deep feature set that correspond to the portion of the query digital image indicated by the image mask.

7. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to: determining a similarity score for a plurality of digital images from the digital image database; and The plurality of digital images are ranked based on the determined similarity scores.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to: receiving a selection of a second query digital image from the user client device; generating a second set of deep features for the second query digital image using the deep neural network based model; generating, based on the second set of deep features of the second query digital image, a multi-query vector representation representing a synthesis of the query digital image and the second query digital image; as well as Similar digital images from within the digital image database to the multiple query vector representations are identified.

9. The non-transitory computer readable medium of claim 8, further comprising instructions that, when executed by the processor, cause the computer device to: receiving, with respect to the second query digital image, a second user selection of at least one of spatial selection, image synthesis, or object counting; and The instructions cause the computer device to generate the multiple query vector representations based on the second user selection by utilizing at least one of the spatial selectivity algorithm, the image synthesis algorithm, or the object counting algorithm.

10. The non-transitory computer-readable medium of claim 8, further comprising instructions that, when executed by the processor, cause the computer device to receive from the user client device: an image mask indicating a portion of the query digital image to emphasize when identifying similar digital images; and A second image mask indicating a portion of the second query digital image to be emphasized when identifying similar digital images.

11. The non-transitory computer-readable medium of claim 10, further comprising instructions which, when executed by the processor, cause the computer device to generate the multi-query vector representation in response to receiving the image mask and the second image mask to emphasize the portion of the query digital image indicated by the image mask and the portion of the second query digital image indicated by the second image mask.

12. A system for matching digital images based on visual similarity in a digital media environment, the digital media environment being used to search for digital images, the system comprising: processor; as well as A non-transitory computer readable medium comprising instructions which, when executed by a processor, cause the system to: receiving from a user client device a user selection of a query digital image and at least one of spatial selectivity, image composition, or object count for identifying similar digital images; Generating a deep feature set for the query digital image using a deep neural network-based model; generating a deep neural network-based representation of the query digital image based on the set of deep features of the query digital image and in accordance with the user selection by utilizing one or more of a spatially selective algorithm, an image synthesis algorithm, or an object counting algorithm; determining a similarity score for a plurality of digital images within a digital image database based on the deep neural network-based representation of the query digital image; as well as Similar digital images to the query digital image are identified from the digital image database based on the deep neural network based representation of the query digital image and also based on the similarity scores for the plurality of digital images.

13. The system of claim 12, wherein the instructions cause the system to determine the similarity score by comparing a depth feature of each of the plurality of digital images to a depth feature of the query digital image.

14. The system of claim 13, further comprising instructions that, when executed by the processor, cause the system to: receiving an image mask from the user client device, the image mask indicating a portion of the query digital image to emphasize when identifying similar digital images; and In response to receiving the image mask, the deep neural network-based representation is generated based on weighting the portion of the query digital image indicated by the image mask.

15. The system of claim 12, further comprising instructions that, when executed by the processor, cause the system to: receiving a selection of a second query digital image from the user client device; generating a second set of deep features for the second query digital image using the deep neural network based model; generating, based on the second set of deep features of the second query digital image, a multi-query vector representation representing a synthesis of the query digital image and the second query digital image; determining a similarity score for a plurality of digital images within a multi-query digital image database based on the deep neural network-based representation of the second query digital image; as well as Similar digital images from within the digital image database to the multiple query vector representations are identified.

16. The system of claim 15, wherein the instructions cause the system to determine the similarity score by comparing deep features of each of the plurality of digital images to deep features of the multiple query vector representations.

17. The system of claim 16, further comprising instructions that, when executed by the processor, cause the system to: Received from the user client device: an image mask indicating a portion of the query digital image to be emphasized when identifying similar digital images; a second image mask indicating a portion of the second query digital image to be further emphasized in identifying similar digital images; A first weight, for applying to the query digital image; as well as a second weight, for applying to the second query digital image; as well as The multiple query vector representation is modified to apply the first weight to the portion of the query digital image indicated by the image mask and to apply the second weight to the portion of the second query digital image indicated by the second image mask.

18. A computer-implemented method for matching digital images based on visual similarity in a digital media environment, the digital media environment being used to search for digital images, the method comprising: receiving, from a user client device, a user selection of a query digital image and at least one of spatial selectivity, image composition, or object count; as well as Step for identifying similar digital images to said query digital image based on said user selection.

19. The computer-implemented method of claim 18, further comprising receiving a second query digital image from the user client device.

20. The computer implemented method of claim 19, further comprising the step of identifying similar digital images to a composite feature vector representing a visual attribute for the query digital image and a visual attribute for the second query digital image.

Images