Synthesis-aware digital image search

Through the dual-stream convolutional neural network to learn the feature representation of the background and prospects in digital image synthesis, the problems of low user interaction efficiency and computing efficiency in the prior art are solved, and efficient digital image synthesis adapted to multi-category image content is realized.

CN110516096BActive Publication Date: 2024-11-05ADOBE INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201910143100.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-22
Filing Date
2019-02-26
Publication Date
2024-11-05
Estimated Expiration
2039-02-26

AI Technical Summary

Technical Problem

The prior art has low user interaction efficiency in digital image synthesis, unable to adapt to the importance of different characteristics, especially in the face of big data, and cannot effectively process millions of digital image sources.

Method used

A dual-stream convolutional neural network (CNN) is used to search synthetically perceived digital images, and the three-tuple learning of digital images are trained to learn the common embedding space, and the feature representation of background and foreground is automatically learned. The machine learning system is used to generate compatibility scores, adapt to multiple categories Image content.

Benefits of technology

It improves the computing efficiency and user interaction experience of digital image synthesis, can handle a large number of digital image sources, adapt to the importance changes of different characteristics, and realizes real-time search and synthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110516096B_ABST
    Figure CN110516096B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to synthetic-aware digital image search. Synthetic-aware digital image search techniques and systems utilizing machine learning are described. In one example, a synthetic-aware image search system employs a two-stream convolutional neural network (CNN) to jointly learn feature embeddings from a foreground digital image that captures a foreground object and a background digital image that captures a background scene. To train a model for the convolutional neural network, triplets of training digital images are used. Each triplet may include a positive foreground digital image and a positive background digital image acquired from the same digital image. The triplet also contains a negative foreground or background digital image that is different from the positive foreground or background digital image that is also included as part of the triplet.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to synthesis-perceptual digital image searching. Background Art

[0002] Digital image synthesis is a fundamental task performed by image processing systems of computing devices as part of digital image editing and graphic design. Digital image synthesis involves combining foreground objects and background scenes from different sources to generate a new composite digital image. However, conventional techniques for performing digital image synthesis are computationally inefficient and frustrating to users due to the inefficiencies in user interaction supported by these conventional techniques. These inefficiencies and user frustrations are exacerbated by the large number of different digital images that can serve as sources for these objects and scenes, which may number in the millions.

[0003] For example, a wide range of features can be used to define the compatibility of a foreground object with a background scene, and the importance of the features can be different based on the content included in the digital image. In one such example, the viewpoint can have greater importance when inserting a foreground object of a car on a background scene of a road. On the other hand, semantic consistency can have greater importance when synthesizing a skier with a snowy mountain. However, conventional techniques focus on a single feature or rely on manual extraction of features to define matching criteria. Therefore, these conventional techniques cannot adapt to different features and the different relative importance of these features when defining matches for different object categories as described above. In addition, these conventional techniques may fail in the face of "big data" because they involve solving millions of digital images that can be used as synthesis sources, one example of which is a stock digital image system accessible via a network. Summary of the invention

[0004] Composition-aware digital image search techniques and systems using machine learning are described. In one example, the composition-aware image search system employs a two-stream convolutional neural network (CNN) to jointly learn feature embeddings from a foreground digital image capturing a foreground object and a background digital image capturing a background scene.

[0005] In order to train the machine learning model of the convolutional neural network, a triplet of training digital images is used. Each triplet includes a positive foreground digital image and a positive background digital image obtained from the same digital image, for example, by using a segmentation mask annotation. The triplet also contains a negative foreground or background digital image, which is different from the positive foreground or background digital image also included as part of the triplet. For example, in order to support the search for foreground objects, positive and negative foreground digital images and positive background digital images are used when training the model. On the other hand, in order to support the search for background scenes, positive and negative background digital images and positive foreground digital images are used when training the model. Then, the triplet of the training digital image is processed by the corresponding background and foreground convolutional neural network (CNN) using a loss function (e.g., triplet loss) to learn a common embedding space. Then, once the common embedding space is learned, the system can use the common embedding space to define the compatibility of the foreground and background digital images with each other.

[0006] This summary introduces some concepts in a simplified form, which will be further described in the following detailed description. Therefore, this summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is described with reference to the accompanying drawings. Entities represented in the drawings may refer to one or more entities, and thus single or multiple forms of the entities may be referenced interchangeably in the discussion.

[0008] Figure 1 is an illustration of an environment in an example implementation that is operable to employ the composition-aware digital image search techniques described herein.

[0009] Figure 2 Depicts a system in an example implementation, which shows in more detail Figure 1 Two-stream system architecture for synthetic-aware image search system.

[0010] Figure 3 Depicts a system in an example implementation, which illustrates in more detail the Figure 2 Two-stream system architecture for synthetic-aware image search system.

[0011] Figure 4 is a flow chart depicting a procedure in an example implementation in which search results are generated based on foreground features learned from a foreground image using machine learning and background features learned from a plurality of candidate digital images.

[0012] Figure 5is a flow chart depicting a procedure in an example implementation in which search results are generated based on background features learned from a background digital image and foreground features learned from a plurality of candidate digital images using machine learning.

[0013] Figure 6 Depicts a system in an example embodiment, which shows in more detail Figure 1 Training of a two-stream system architecture for a synthetic-aware image search system 118.

[0014] Figure 7 Depicts a system in an example embodiment, which illustrates in more detail the Figure 1 Two-stream system architecture for synthetic-aware image search system.

[0015] Figure 8 An example of training digital images generated for training a synthetic-aware image search system is depicted.

[0016] Fig. 9 Depicted is an example implementation in which the Figure 1 The synthetic perceptual image search system 118 generates additional positive training digital images.

[0017] Fig.10 Depicted is an example implementation in which the Figure 1 The synthetic-aware image search system generates additional negative training digital images.

[0018] Fig.11 A procedure is depicted in an example implementation in which training digital images are generated for use in digital image synthesis.

[0019] Fig.12 An example system including various components of an example device is illustrated, which can be implemented as described in reference Figure 1-Figure 11 Any type of computing device described and / or utilized to implement embodiments of the techniques described herein. DETAILED DESCRIPTION

[0020] Overview

[0021] Conventional techniques for synthesizing digital images typically rely on manually extracted features and defined matching criteria, or rely on a single characteristic that defines when a foreground object is compatible with a background scene. Thus, conventional techniques fail to account for the wide variety of characteristics that may be used to define this compatibility, the importance of which may vary for different categories of digital images (i.e., the content captured by the image).

[0022] Thus, synthetic-aware digital image search techniques and systems that utilize machine learning are described. Thus, instead of manually designing matching criteria or hand-engineering features, machine learning can be used (e.g., by using corresponding convolutional neural networks) to automatically and accurately learn feature representations through a synthetic-aware image search system for background scenes and foreground objects. For example, the synthetic-aware image search system can employ a two-stream convolutional neural network (CNN) to jointly learn feature embeddings from a foreground digital image that captures a foreground object and a background digital image that captures a background scene.

[0023] To train a model for a convolutional neural network, triplets of training digital images are used from a dataset with segmentation mask annotations to learn a common embedding space. For example, each triplet may include a positive foreground digital image and a positive background digital image obtained from the same digital image (e.g., by using a segmentation mask annotation). The triplet also contains a negative foreground or background digital image that is different from the positive foreground or background digital image also included as part of the triplet. For example, to support a search for foreground objects, positive and negative foreground digital images and positive background digital images are used when training the model. On the other hand, to support a search for background scenes, positive and negative background digital images and positive foreground digital images are used when training the model. Similarity (positive compatibility) or dissimilarity (i.e., negative compatibility) can be based on features that can be used to define the compatibility of digital images with each other, such as semantics, viewpoint, style, color, etc.

[0024] The triplets of training digital images are then processed by corresponding background and foreground convolutional neural networks (CNNs) jointly using a loss function (e.g., a triplet loss) to learn a common embedding space, i.e., a machine learning model for training convolutional neural networks. Once learned, this common embedding space can then be used by the system to define the compatibility of foreground and background digital images with each other. For example, this can be performed by calculating a score based on the cosine similarity of feature vectors for background features and foreground features learned by the corresponding convolutional neural networks of the two-stream system. Category data can also be embedded as part of the background and foreground features to further guide compatibility, for example by using category features learned from the text of the category data using word2vec.

[0025] In this way, the generation of features from both background and foreground can be used to address a wide range of characteristics that can be used to define compatibility, thereby overcoming the limitations of conventional techniques. In addition, these techniques can be used to address the millions of available digital images that can be used as synthesis sources. For example, background features can be calculated offline and then used in real time with foreground features (which are also generated in real time) to perform synthesis-aware searches that are not possible using conventional manually defined techniques.

[0026] The technology can also be adopted by the synthetic perception image search system to expand the multiple training digital images that can be used to train background and foreground convolutional neural networks. For example, the system can employ a training data generation module to generate additional examples of positive or negative background or foreground digital images to be used for training. In one example, this is performed by replacing the positive foreground digital image associated with the positive background digital image with a similar positive foreground digital image obtained from another digital image. In another example, this is performed by replacing the positive background digital image associated with the positive foreground digital image with a similar positive background digital image obtained from another digital image. Similar techniques can also be used to increase multiple negative training digital images, such as by replacing the positive foreground or background digital image with a different foreground or background digital image. In this way, even by using a training data set with a limited number of annotated (e.g., masked) examples, the training accuracy as part of machine learning can be improved. Further discussion of these and other examples is included in the following sections and is shown in the corresponding figures.

[0027] In the following discussion, an example environment is described in which the techniques described herein may be employed. Example programs are also described that may be executed in the example environment as well as other environments. Thus, execution of the example programs is not limited to the example environment, and the example environment is not limited to execution of the example programs.

[0028] Example Environment

[0029] Figure 1 1 is an illustration of a digital media environment 100 in an example implementation that is operable to employ the composition-aware digital image search techniques described herein. The illustrated environment 100 includes a service provider system 102 and a client device 104, which can be configured in various ways. The computing devices that implement the service provider system 102 and the client device 104 can be configured in various ways.

[0030] For example, the computing device may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet computer or a mobile phone, as shown), etc. Thus, the computing device may range from a full resource device with a large amount of memory and processor resources (e.g., a personal computer, a game console) to a low resource device with limited memory and / or processing resources (e.g., a mobile device). In addition, the computing device may represent a plurality of different devices, such as a plurality of servers used to implement the service provider system 102.

[0031] The client device 104 is illustrated as including an image processing system 108. The image processing system 108 is implemented at least in part in hardware of the client device 104 to process and transform a digital image 110, which is illustrated as being maintained in a storage device 112 of the client device 104. Such processing includes creation of the digital image 110, modification of the digital image 110, and rendering of the digital image 110 in a user interface 114 for output, for example, by a display device 116. Although illustrated as being implemented locally on a computing device, the functionality of the image processing system may also be implemented in whole or in part by functionality available via the network 106, such as as part of a network service or "in the cloud."

[0032] An example of the functionality included by the image processing system 108 to process digital images includes digital image synthesis. Digital image synthesis involves combining foreground objects and background scenes from different sources to generate a new composite digital image. For example, a user may wish to add a foreground object to a background scene, or change the background scene of a foreground object.

[0033] To do so in the illustrated example, the image processing system 108 accesses the synthesis-aware image search system 118 of the service provider system 102 via the network 106. Other examples are also contemplated in which the synthesis-aware image search system 118 is implemented locally on the client device 104 used to perform the synthesis. The access is used to perform a search of digital images 120 maintained in the storage device 122 that are compatible with the digital images 110 of the client device 104.

[0034] As previously described, various features may be used to define the compatibility of foreground and background images with each other, and the importance of the features may change based on the category to which the image belongs. For example, when inserting a foreground object of a car on a background scene of a road, the viewpoint may have greater importance, while when synthesizing a skier with a snowy mountain, semantic consistency may have greater importance. Therefore, the synthesis-aware image search system 118 includes a background feature machine learning system 124 and a foreground feature machine learning system 126, which are configured to generate background and foreground features, respectively. The background and foreground features can be used to determine the compatibility of the foreground image with the background image. In one embodiment, this can also be assisted by using a category feature machine learning system 128, which can be used to learn category features from category data provided with the foreground and background images. For example, the category data can define categories, which define "what" is included in the foreground and background digital images, thereby facilitating the search, as further described below. In one example, this is performed for eight categories that appear frequently and are widely used in image synthesis: "person", "car", "boat", "dog", "plant", "bottle", "chair" and "painting".

[0035] In this manner, background digital image 130 may be used as a basis for generating image feature data 132 including background features 134 for determining compatibility with foreground digital image 120. Likewise, foreground digital image 136 may be used to generate image feature data 138 having foreground features 140 for determining compatibility with the background digital image.

[0036] By using the background feature machine learning system 124 and the foreground feature machine learning system 126, the learned feature representations can encode rich information specifically for image synthesis and can be adapted to a variety of image content. In addition, the synthesis-aware image search system 118 can address multiple foreground and / or background categories in a single framework, thereby improving the computational efficiency and operation of computing devices that employ these techniques. In this way, the synthesis-aware image search system 118 can address a variety of categories, even if not specifically trained by the system, which is not possible using conventional manual or single-category techniques.

[0037] Typically, the functions, features, and concepts described in the examples above and below can be employed in the context of the example programs described in this section. In addition, the functions, features, and concepts described in the different figures and examples herein can be interchangeable with each other and are not limited to the implementation methods in the context of a specific figure or program. In addition, the boxes associated with the different representative programs and corresponding figures herein can be applied together and / or combined in different ways. Therefore, the individual functions, features, and concepts described in the different example environments, devices, components, figures, and programs herein can be used in any suitable combination and are not limited to the specific combinations represented by the examples listed in this specification.

[0038] Synthetic Perception Digital Image Search System Architecture

[0039] Figure 2 The system 200 in an example embodiment is depicted, which shows in more detail Figure 1 A two-stream system architecture of the synthetic-aware image search system 118. Figure 3 Depicted is a system 300 in an example implementation, which illustrates in more detail the Figure 1 A two-stream system architecture of the synthetic-aware image search system 118. Figure 4 A procedure 400 is depicted in an example implementation in which search results are generated based on foreground features learned from a foreground image using machine learning and background features learned from a plurality of candidate digital images. A procedure 500 is depicted in an example implementation in which search results are generated based on background features learned from a background digital image using machine learning and foreground features learned from a plurality of candidate digital images.

[0040] The following discussion describes techniques that can be implemented using the described systems and devices. Aspects of the procedures can be implemented in hardware, firmware, software, or a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the order shown for performing the operations of the various blocks. In portions of the following discussion, references will be made interchangeably to Figure 1-Figure 4 .

[0041] like Figure 2 As shown, the synthetic-aware image search system 118 is formed as a two-stream feature embedding network configured to determine the compatibility of a background image with a foreground ground image. The first stream is represented by a background feature machine learning system 124, and the second stream is represented by a foreground feature machine learning system 126. The background feature machine learning system 124 and the foreground feature machine learning system 126 employ corresponding convolutional neural networks 206, 208 to learn background features 210 and foreground features 212, respectively.

[0042] The convolutional neural networks 206, 208 are formed for node layers (i.e., neurons) and include an input layer, an output layer, and one or more hidden layers. The hidden layers may include convolutional layers, pooling layers, fully connected layers, and / or normalization layers. The convolutional neural networks 206, 208 are trained to learn a common embedding space, which is then used to determine the compatibility of a background image with a foreground image, such as using cosine similarity. Further discussion of model training is included in the synthetic perception training data generation and machine learning section of the following discussion.

[0043] In the illustrated example, the background feature machine learning system 124 takes a background image as input. A portion of the digital image that is to receive the foreground object is filled in by the system. To do this, the background feature machine learning system 124 calculates the color average of the colors of the background image and uses the color average to fill in a rectangle with the height and width of the foreground object to be composited with the background. In this way, the filled portion defines the size, aspect ratio, and position in the background scene for receiving the foreground object.

[0044] The foreground feature machine learning system 126 accepts the foreground digital image 204 as input to generate foreground features 212. The foreground digital image 204 in this example includes a foreground object with a solid background (e.g., a white background). Examples of these foreground digital images can be obtained from a stock image system (e.g., Stock), which is abundant and preferred by creative professionals, while being difficult to search using conventional techniques such as keyword searching.

[0045] The background features 210 and the foreground features 212 are provided to a score calculation module 214 to calculate a score 216 indicating the compatibility of the background and foreground digital images. The score 216 is based on how “close” the background features 210 and the foreground features 212 are to each other in a common embedding space learned by the convolutional neural networks 208, 208 (e.g., using cosine similarity). The search module 218 then uses these scores to sort the candidate digital images 120 in the storage device 122 to generate search results 220 indicating the compatibility of the digital images with each other. This can be performed to locate a foreground digital image 204 for a corresponding background digital image 202, or to locate a background digital image 202 for a foreground digital image 204.

[0046] Figure 3 An example system 300 is depicted, wherein Figure 2 The synthetic-aware image search system 118 is adapted to resolve the category data 302. In this example, the synthetic-aware image search system 118 is configured to resolve multiple image categories in a single framework, which is not possible using conventional techniques. In this way, the synthetic-aware image search system 118 can resolve differences in the importance of features when determining the compatibility of foreground and background digital images for various categories, even without being specifically trained by the system.

[0047] To do this in the illustrated example, the category data 302 is received by a category feature machine learning system 304, which is processed via a neural network 306 to generate category features 308. For example, the neural network 306 can employ word2vec technology, which is a set of related machine learning models for generating category features as word embeddings. These machine learning models can be implemented as a two-layer neural network that is trained to reconstruct the language context of a word.

[0048] The neural network 306 is trained using a text corpus to learn a vector space that is typically several hundred dimensions, where each unique word in the corpus is assigned a corresponding vector in the vector space. Thus, the word vectors are located in the vector space so that words that share a common context in the corpus are located very close to each other in the space. In this way, a wide range of category data 302 can be employed to describe the characteristics of background digital images 202 and foreground digital images 204. This can even be used to account for category data 302 that has not been specifically trained by the synthetic-aware image search system 118 by determining the proximity of untrained words in the vector space to words used as part of the training.

[0049] In order to combine the category features by the synthetic perception image search system 118, the category features 308 are encoded as part of the background features 210 and the foreground features 212. To this end, the multi-modal compact bilinear pooling (MCB) module 310 is used in the illustrated example to obtain the outer product of two vectors (e.g., background features 210 and category features 308; or foreground features 212 and category features 308) to form a combination, although other techniques are also contemplated. The feature conversion modules 314, 316 are then used to employ inner products and compact bilinear pooling and light calculation CNN to generate a score through the use of the score calculation module 214 using a triplet loss function.

[0050] In one implementation example, a ResNet50 deep neural network is used to learn features (e.g., up to the "pool5" layer) as initial weights, after which global average pooling is performed to obtain a feature map of size "1x 1x 2048". Although the background stream and foreground stream of the background feature machine learning system 124 and the foreground feature machine learning system 126 are initialized with the same weights from ResNet50, different information may be encoded after learning these streams. For example, the background feature machine learning system 124 may be configured to focus on the context of the background scene, while the foreground feature machine learning system 126 focuses on object-oriented features.

[0051] In order to learn the category feature 308 as a category-specific feature map, the category feature machine learning system 304 adopts the word2vec model as described above to extract a 300-dimensional vector as the category feature 308. After several convolutional layers, the category feature 308 is then fused with the background feature 210 and the foreground feature 212 in each individual MCB module 310, 312. The lightweight feature conversion module 314, 316 includes a convolutional layer, an element-wise signed square root layer, and a weighted feature transformation module 314, 316. In addition to the example l2 normalization operation, lightweight feature conversion modules 314, 316 are further added to the system. This is used to generate unit feature vectors for the background and foreground, respectively, which encode the image content and category data 302 from the background digital image 202 and the foreground digital image 204.

[0052] This can be used to support a variety of usage scenarios.In a first example, a creative professional may interact with the image processing system 108 of a client device 104 having a background scene of a background digital image 130 and wishes to locate a foreground object from a foreground digital image to be composited with the background scene.

[0053] Thus, the background digital image 202 is provided to the synthetic-perceptual image search system 118. The background feature machine learning system 124 is then employed to extract background features 210 from a digital image (e.g., the background digital image 202) using machine learning (block 402), such as via a convolutional neural network 206. This can be performed "online" in real time, within seconds, upon receipt of the background digital image 202.

[0054] The synthetic-aware image search system 118 also employs the foreground feature machine learning system 126 to extract foreground features 212 from the plurality of candidate digital images 120 using machine learning (block 404). In this example, the foreground feature machine learning system 126 may perform this extraction "offline" to extract the foreground features 212 from a repository of candidate digital images of foreground objects. The foreground features 212 may then be stored in a storage device to support real-time searching based on the background features 210 in this example, which is not possible using conventional techniques.

[0055] exist Figure 3 In the example system 300 of FIG. 1 , the synthetic-aware image search system 118 is further configured to include a category feature machine learning system 304 to generate category features from category data 302 including text (block 406). For example, the category data 302 may provide a semantic context of the characteristics of the background and foreground digital images to be used as a basis for determining compatibility. For example, the category features 308 may be embedded as part of the background features 210 and the foreground features 212 by the corresponding MCB 310, 312 modules and processed using the corresponding feature conversion modules 314, 316.

[0056] Then, the score calculation module 214 is implemented to calculate the score (box 408) by feature embedding based at least in part on the background features extracted from the digital image and the foreground features extracted from the plurality of candidate digital images, which is then used to output the search results 220 (box 410) based on the calculated score. The score 216 indicates the relative proximity of the background and foreground features to each other in the common embedding space. In this way, the score 216 can be used to define the relative amount of compatibility of the background image with each candidate digital image that will serve as the foreground image. These scores can then be used to rank the candidate digital images to be output as part of the search results 220. In addition, therefore, the use of online and offline feature generation can support the real-time reception and generation of the search results 220, which is impossible to achieve using conventional techniques. These techniques can also be used to locate the background scene for the foreground object, as further described in the following examples.

[0057] Figure 5A procedure 500 is depicted in an example implementation in which search results are generated based on background features learned from a background digital image using machine learning and foreground features learned from a plurality of candidate digital images. In this example, a creative professional may be working with an image processing system 108 of a client device 104 having a foreground object of a foreground digital image 136 and wishes to locate a compatible background scene for the background digital image to be synthesized with the foreground object. Thus, the foreground digital image 204 is provided to a synthesis-aware image search system 118.

[0058] Then, foreground feature machine learning system 126 is employed to extract foreground features 212 from a digital image (e.g., foreground digital image 204) using machine learning, such as via convolutional neural network 208 (block 502). This can be performed "online" in real time as foreground digital image 204 is received.

[0059] The synthetic-aware image search system 118 also employs the background feature machine learning system 124 to extract background features 210 from the plurality of candidate digital images 120 using machine learning (block 504). In this example, the background feature machine learning system 124 may perform this extraction “offline” to extract the background features 210 from a repository of candidate digital images 120 of background scenes. The background features 210 may then be stored in a storage device to support real-time searching based on the foreground features 212 in this example, which is not possible using conventional techniques.

[0060] exist Figure 3 In the example system 300 of FIG. 5 , the synthetic-aware image search system 118 is further configured to include a category feature machine learning system 304 to generate category features from category data 302 including text (block 506). For example, the category data 302 may provide a semantic context of the characteristics of the background and foreground digital images to be used as a basis for determining compatibility. For example, the category features 308 may be embedded as part of the background features 210 and the foreground features 212 by the corresponding MCB 310, 312 modules and processed using the corresponding feature conversion modules 314, 316.

[0061] Then, the score calculation module 214 is also implemented to calculate the score (box 508) by feature embedding based at least in part on the background features extracted from the digital image and the foreground features extracted from the plurality of candidate digital images, which is then used to output the search results 220 (box 510) based on the calculated score. The score 216 indicates the relative proximity of the background and foreground features to each other in the common embedding space, and thus the score 216 can be used to define the relative amount of compatibility of the foreground image with each candidate digital image to be used as the background image. These scores can then be used to rank the candidate digital images for output as part of the search results. In addition, therefore, users of online and offline feature generation can support the reception and generation of the search results 220 in real time as described above, which is not possible using conventional techniques.

[0062] Thus, synthetic-aware digital image search techniques and systems are described that overcome challenges of conventional techniques. Thus, instead of manually designing matching criteria or hand-engineering features, machine learning can be used, such as by using corresponding convolutional neural networks, to automatically and accurately learn feature representations through the synthetic-aware image search system 118 for background scenes and foreground objects. Training of the synthetic-aware image search system 118 is described in the following sections and illustrated in the corresponding figures.

[0063] Synthetic Perception Training Data Generation and Machine Learning

[0064] Figure 6 Depicted is a system 600 in an example implementation, which illustrates in more detail the Figure 1 Training of a two-stream system architecture for a synthetic-aware image search system 118. Figure 7 Depicted is a system 700 in an example implementation, which illustrates in more detail the Figure 1 A two-stream system architecture of the synthetic-aware image search system 118. Figure 8 An example 800 of training digital images generated for use in training the synthetic-aware image search system 118 is depicted. Fig. 9 An example implementation 900 is depicted in which Figure 1 The synthetic perceptual image search system 118 generates additional positive training digital images. Fig.10 An example implementation 1000 is depicted in which Figure 1 The synthetic perceptual image search system 118 generates additional negative training digital images. Fig.11 A procedure 1100 is depicted in an example implementation in which training digital images are generated for use in digital image synthesis.

[0065] The following discussion describes techniques that can be implemented using the described systems and devices. Aspects of the procedures can be implemented in hardware, firmware, software, or a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the order shown for performing the operations of the respective blocks. In portions of the following discussion, references will be made interchangeably to Figure 6-Figure 11 .

[0066] The triplets of training digital images are used to train the background feature machine learning system 124 and the foreground feature machine learning system 126. Figure 6 In the illustrated example of , this is performed using a positive background digital image 602, a positive foreground digital image 604, and a negative foreground digital image 606. As previously described, this is used to generate corresponding background features 608 and foreground features 610, and thereby the loss function 612 is used to generate a triplet loss 614 to train the background feature machine learning system 124 and the foreground feature machine learning system 126.

[0067] Category data 702 can also be used as part of the training, such as Figure 7 . In this example, the category data 702 is processed as training data by the neural network 306 of the category feature machine learning system 304 to generate category features 708, such as vector representations of text using word2vec. The category features 708 are embedded as part of the background features 608 and foreground features 610 using the MCB 310, 312 modules and processed using the feature conversion modules 314, 316 as described above. The loss function 612 is again used to calculate the triplet loss to train the background feature machine learning system 124 and the foreground feature machine learning system 126.

[0068] The accuracy of training the background feature machine learning system 124 and the foreground feature machine learning system 126 for image synthesis can be increased by increasing the number of training digital images used as part of the training. However, a set of training digital images is not available from conventional systems to support synthesis-aware image search. Collecting such a training set using conventional techniques is resource intensive because manually labeling many pairs of background and foreground digital images is time consuming and also requires expertise in image synthesis and digital image editing to determine whether the two are compatible.

[0069] Therefore, in this example, the synthetic-aware image search system 118 is configured to automatically generate training digital images without user intervention. To this end, digital images are obtained from publicly available datasets containing object instance segmentation masks, examples of which include MS-COCO, PASCAL VOC2012, and ADE20K. Using these mask annotations, the training data generation module 802 decomposes each digital image into a background scene and a foreground object. Figure 8 , where an original digital image 804 is used to generate a positive background digital image 806 and a positive foreground digital image 808. Because the positive background digital image 806 and the positive foreground digital image 808 are generated from the same original digital image 804, these images are compatible for synthesis and are generally more suitable than any other possible candidate pair.

[0070] Therefore, to form a triplet of training digital images, the positive background digital image 806 is considered as an anchor and the positive foreground digital image 808 is considered as a positive sample. Foreground objects are randomly sampled from any other image as negative samples, which are illustrated as negative foreground digital images 810. Category data can also be used as part of this training. In one example, this is performed for eight categories that appear frequently and are widely used in image synthesis: "person", "car", "boat", "dog", "plant", "bottle", "chair", and "painting".

[0071] In practice, the foreground object is usually not included in the background digital image that will be used as the basis for locating the foreground object in the foreground digital image. To simulate this situation in training, the area 812 where the foreground object is taken from the original digital image 804 is filled to remove the foreground object from the background scene. In one example, the area 812 is filled with a color that is calculated as the color average of the color of the background digital image from which the foreground digital image is extracted.

[0072] By including the fill region 812 in the positive background digital image 806, the learned background features 608 can be responsive to the location, size, and / or aspect ratio of the foreground object to be inserted as part of determining the compatibility of the foreground and background digital images. For example, when inserting a person on a lawn, a tall rectangle means that the user may want a standing person, while a wide rectangle may indicate a sitting person. This constraint is not strictly enforced because there may be limitations on user accuracy.

[0073] In order to address these limitations, training data generation module 802 can implement data enhancement processing, which relaxes the size and ratio constraints between the foreground and background digital images of the pairing to a limited degree. For background enhancement, random filling is added to the bounding box, wherein the maximum possible filling space is set to half the width and height of the bounding box. The filled area is also filled with an average color value. Similarly, for foreground enhancement, random filling is added to fill the filled area with white filling, for example. For the negative foreground digital image in the triplet, it is randomly selected from another image by similar enhancement techniques. Although some foreground objects that are actually compatible with the background can be selected, the foreground from the same image still has a greater degree of compatibility, so the difference between the positive example and the negative example still holds.

[0074] Although the training data generation module 802 can quickly and efficiently generate pairs of positive foregrounds and backgrounds from the same image, for digital images with "m" number available, the number of positive samples is significantly less than the number of negative samples (e.g., "m" relative to "m(m-1)"). The severe imbalance in the number of training samples, coupled with the noise in the negative pair sampling, where some compatible foregrounds are mishandled as negative samples, poses a challenge. To address this issue, a triple expansion strategy can be adopted by the training data generation module 802 to generate additional positive samples, and can also be executed to generate additional negative samples.

[0075] For example, Fig. 9 Depicted is the generation of a positive foreground digital image 808 and a positive background digital image 806 from an original digital image 804 as described above. The training data generation module 802 then employs matching criteria to find similar foreground digital images 902 and / or background digital images 904. Examples of matching criteria include semantic context and shape information. For shape information, after alignment based on mask centers, the intersection over union (IoU) score of the two foreground masks is calculated. In this example, the foreground with a higher IoU score is considered to have greater similarity.

[0076] For semantic context information, the foreground digital image is filled with the original background digital image, and semantic features are extracted using machine learning through a neural network trained for image classification (e.g., using ResNet50). Similar foreground digital images are then retrieved by comparing the Euclidean "l2" distance of the extracted features in a common embedding space 902. This technique produces results with greater consistency than other techniques that rely on feature extraction with a "pure" background, such as white pixels.

[0077] In fact, foreground objects with rigid shapes have increased sensitivity to the viewpoint used to view the object. Therefore, shape information has a higher effectiveness in finding similar foreground digital images. In addition, foreground objects with more diverse appearances can vary depending on the scene, so semantic context information has increased effectiveness in finding similar foreground digital images. Therefore, in the example above, shape information is used to locate positive foreground images for "bottle", "car", "chair", and "painting", and semantic context information is used to locate similar foreground images for "boat", "dog", "person", and "plant".

[0078] Thus, given a positive foreground digital image 808 and its corresponding positive background digital image 806 from the same original digital image 804, the top "N" similar foreground digital images 902 are located from the digital image 120 by the training data generation module 802. These similar foreground digital images 902 are considered compatible foregrounds for the positive background digital image 806, for example as new triplets of training digital images. In this way, the number of positive training pairs can be increased, and the noise in the negative pair sampling can also be reduced. This can also be used to replace the positive background digital image 806 with a similar background digital image 904, which, when combined with the positive foreground digital image 808, is also used to increase the number of triplets of training digital images.

[0079] Fig.10 An example implementation 1000 is depicted in which Figure 1 The synthetic perceptual image search system 118 generates additional negative training digital images. The training data generation module 802 in this example generates additional negative samples to be used as part of the triplet of training digital images. To this end, the training data generation module 802 locates dissimilar foreground or background digital images 1002, 1004 based on the criteria described above. These criteria can include using shape information or feature extraction by comparing the Euclidean distance (i.e., "l2" distance) of the features extracted in the common embedding space.

[0080] Will not refer to Fig.10 , wherein a positive foreground digital image 808 and a background digital image are extracted from a single digital image (block 1102). For example, the original digital image 804 may include mask data that marks individual pixels of the digital image. The markings can then be used to extract the foreground object as a foreground digital image that is separated from the background scene of the background digital image.

[0081] Positive background image 806 is then generated by filling region 812 in the background digital image from which positive foreground digital image 808 was extracted (block 1104). For example, region 812 may be filled with a color calculated as the color average of the color values ​​of pixels in positive background digital image 806 that are "outside" of region 812. Region 812 thus provides a size, shape, and location that can be used to guide the search for a foreground digital image.

[0082] A negative foreground digital image is also obtained (block 1106 ). To this end, the training data generation module 802 randomly samples foreground objects from the digital image 120 repository as negative samples, which are illustrated as negative foreground digital images 810 .

[0083] Category features may also be generated from category data including text (block 1108). The category features are used to guide the search by learning vector representations of the text using a neural network such as word2vec. In one example, this is performed for eight categories that appear frequently and are widely used in image synthesis: 'person', 'car', 'boat', 'dog', 'plant', 'bottle', 'chair', and 'painting'. In this way, the category data can be used to 'guide' the search.

[0084] The background feature machine learning system 124 and the foreground feature machine learning system 126 are trained jointly using a loss function (block 1110) based on the positive foreground digital image 808, the positive background digital image 806, and the negative foreground digital image 810. This may also include training using class features embedded as part of the background and foreground features extracted from the image.

[0085] Therefore, the machine learning models of the convolutional neural networks 206, 208 of the background feature machine learning system 124 and the foreground feature machine learning system 126 are trained using training data, which is configured as triplets of digital images. The triplets are constructed to include positive and negative samples of the image type being searched (e.g., foreground or background) and positive examples of alternative image types that are not being searched. For example, for a foreground search, a triple includes: a background image as an anchor, a compatible foreground as a positive sample, and an incompatible foreground as a negative sample. On the other hand, for a background search, a triple includes: a foreground image as an anchor, a compatible background as a positive sample, and an incompatible background as a negative sample.

[0086] The neural network is trained with triplet loss and feature similarity (e.g., feature similarity between background anchors and positive foreground) is enforced to be closer to feature similarity between anchors and negative samples. Because feature vectors have matching unit lengths after Euclidean "l2" normalization, their similarity can be calculated as squared Euclidean "l2" distance. In order to encourage the ability to distinguish between positive and negative pairs, a positive margin "α" is introduced for class "i". i For convenience, feature extraction, multi-mode compact bilinear pooling, and “l2” normalization can be grouped into the following operations:

[0087]

[0088] in and are the operations for category “i” in the background and foreground streams, respectively. The value “B i "and represents the background digit image and its associated positive (i.e., compatible) and negative foreground objects. During training, the following loss function Minimized:

[0089]

[0090] The design exhibits higher effectiveness than features shared across multiple categories, which cannot encode sufficient category-specific information. These techniques also exhibit higher computational efficiency than independently learning separate feature representations dedicated to each category. The offline and online feature extraction techniques described above can also support real-time operations (e.g., in seconds) on millions of digital images, thereby supporting the richness of user interactions and improved operational efficiency of computing devices that implement these techniques.

[0091] Example Systems and Devices

[0092] Fig.12 An example system is illustrated generally at 1200, which includes an example computing device 1202, which represents one or more computing systems and / or devices that may implement the various techniques described herein. This is illustrated by including the synthetic-aware image search system 118. The computing device 1202 may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0093] The example computing device 1202 shown includes a processing system 1204, one or more computer-readable storage media 1206, and one or more I / O interfaces 1208 coupled to each other in communication. Although not shown, the computing device 1202 may also include a system bus or other data and command transmission system that couples various components to each other. The system bus may include any one or combination of different bus structures (such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of the various bus architectures). Various other examples are also contemplated, such as control and data lines.

[0094] Processing system 1204 represents the functionality of performing one or more operations using hardware. Thus, processing system 1204 is illustrated as including hardware elements 1210, which may be configured as processors, functional blocks, etc. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 1210 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may include (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In this context, processor executable instructions may be electronic executable instructions.

[0095] Computer-readable storage media 1206 is illustrated as including memory / storage 1212. Memory / storage 1212 represents memory / storage capacity associated with one or more computer-readable media. Memory / storage 1212 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 1212 may include fixed media (e.g., RAM, ROM, fixed hard drives, etc.) and removable media (e.g., flash memory, removable hard drives, optical disks, etc.). Computer-readable storage media 1206 may be configured in various other ways, as further described below.

[0096] Input / output interface(s) 1208 represent functionality that allows a user to input commands and information to computing device 1202, and also allows information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., which may employ visible wavelengths or invisible wavelengths such as infrared frequencies to recognize motion as gestures that do not involve touch), and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a network card, a tactile response device, and the like. Thus, computing device 1202 may be configured in a variety of ways, as further described below, to support user interaction.

[0097] Various techniques may be described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on various commercial computing platforms with various processors.

[0098] Implementations of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include a variety of media that may be accessed by computing device 1202. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0099] "Computer-readable storage media" may refer to media and / or devices capable of storing information persistently and / or non-temporarily, rather than just signal transmissions, carrier waves, or signals themselves. Therefore, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media, and / or storage devices implemented in a method or technology suitable for storing information such as computer-readable instructions, data structures, program modules, logical elements / circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0100] "Computer-readable signal media" may refer to signal-bearing media that is configured to transmit instructions to the hardware of the computing device 1202, such as via a network. Signal media may typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0101] As previously described, hardware elements 1210 and computer-readable storage media 1206 represent modules, programmable device logic, and / or fixed device logic implemented in hardware that may be employed in some embodiments to implement at least some aspects of the techniques described herein, such as for executing one or more instructions. Hardware may include components of an integrated circuit or system on a chip, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations of silicon or other hardware. In this context, the hardware may operate as a processing device that performs program tasks defined by instructions and / or logic implemented by the hardware, as well as hardware (e.g., the previously described computer-readable storage medium) for storing instructions for execution.

[0102] The aforementioned combination can also be used to implement the various technologies described herein. Therefore, software, hardware or executable modules can be implemented as one or more instructions and / or logic implemented on some form of computer-readable storage medium and / or by one or more hardware elements 1210. Computing device 1202 can be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, the implementation of the module that can be executed as software by computing device 1202 can be at least partially implemented in hardware, such as by using computer-readable storage media and / or hardware elements 1210 of processing system 1204. Instructions / or functions can be executed / operated by one or more products (e.g., one or more computing devices 1202 and / or processing systems 1204) to implement the techniques, modules and examples described herein.

[0103] The techniques described herein can be supported by various configurations of computing device 1202 and are not limited to the specific examples of the techniques described herein. The functionality can also be implemented in whole or in part by using a distributed system, such as on a "cloud" 1214 through a platform 1216 as described below.

[0104] The cloud 1214 includes and / or represents a platform 1216 for resources 1218. The platform 1216 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of the cloud 1214. The resources 1218 may include applications and / or data that may be used when executing computer processing on a server remote from the computing device 1202. The resources 1218 may also include services provided over the Internet and / or through a subscriber network such as a cellular or Wi-Fi network.

[0105] The platform 1216 can abstract resources and functionality to connect the computing device 1202 with other computing devices. The platform 1216 can also be used to abstract the scaling of resources to provide a corresponding level of scale to meet the demand for resources 1218 implemented via the platform 1216. Therefore, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 1200. For example, the functionality can be implemented partially on the computing device 1202 and via the platform 1216 that abstracts the functionality of the cloud 1214.

[0106] in conclusion

[0107] Although the invention has been described in language specific to structural features and / or methodological acts, it should be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.

Claims

1. A system for searching digital images in a digital media synthesis-aware digital image environment, include: a background feature machine learning system, implemented at least in part in hardware of at least one computing device, to receive a digital image having a background and extract background features from the background of the digital image using a first neural network as part of machine learning; a foreground feature machine learning system, implemented at least in part in hardware of the at least one computing device, to receive a plurality of candidate digital images having foregrounds and to extract foreground features from the foregrounds using a second neural network as part of the machine learning; a score calculation module, implemented at least in part in hardware of the at least one computing device, to calculate a score by feature embedding based on the extracted background features from the digital image and the extracted foreground features from the plurality of candidate digital images; as well as A search module is implemented at least in part in the hardware of the at least one computing device to output search results including at least one of the candidate digital images based on the calculated score.

2. The system of claim 1 , wherein the background feature machine learning system is implemented by the first neural network as a first convolutional neural network, and the foreground feature machine learning system is implemented by the second neural network as a second convolutional neural network.

3. The system according to claim 1, further comprising: include: A categorical feature machine learning system is implemented at least in part in hardware of the at least one computing device to generate categorical features from categorical data including text that describes the digital image.

4. The system of claim 3, wherein the category feature is a vector representation of the text formed using machine learning. 5 . The system of claim 3 , wherein the category features are embedded into the background features and the foreground features and are used by the score calculation module to calculate the score.

6. The system of claim 5, wherein the embedding is performed using corresponding multi-mode compact bilinear pooling (MCB) modules.

7. A method implemented by a computing device in a digital media environment, the method include: Extracting a positive foreground digital image and a background digital image from a single digital image by the computing device; Generating, by the computing device, a positive background digital image by filling an area in the background digital image, the positive foreground digital image being extracted from the background digital image, such that features of the positive foreground digital image are compatible with features of the positive background digital image; obtaining, by the computing device, a negative foreground digital image, wherein features from the negative foreground digital image are incompatible with features of the positive background digital image; as well as The computing device jointly uses a loss function to train a background feature machine learning system and a foreground feature machine learning system based on the positive foreground digital image, the positive background digital image, and the negative foreground digital image.

8. The method according to claim 7, further comprising: include: Generating a categorical feature from categorical data as a vector representation of text formed using machine learning, the categorical data including the text, and wherein the training is based at least in part on the categorical feature.

9. The method of claim 7, wherein the filling comprises forming a rectangle having a size, aspect ratio, or position corresponding to the foreground digital image disposed in the digital image.

10. The method of claim 7, further comprising calculating a color average of colors of the background digital image from which the foreground digital image is extracted, and wherein the filling fills the region using the color average. The method according to claim 7 , wherein the loss function is a triplet loss function.

12. The method of claim 7, further comprising retrieving an additional positive foreground digital image based on the positive foreground digital image.

13. The method of claim 12, wherein the retrieving is based at least in part on a semantic context or shape of the positive foreground digital image.

14. The method of claim 12, wherein the training is performed based on the additional positive foreground digital image and the positive background digital image, jointly using the loss function.

15. The method of claim 7, further comprising retrieving an additional positive background digital image based on the positive background digital image.

16. The method of claim 15, wherein the retrieving is based at least in part on a semantic context or shape of the positive background digital image.

17. The method of claim 15, wherein the training is performed based on the additional positive background digital image and the positive foreground digital image, jointly using the loss function.

18. A system for searching digital images in a digital media synthesis-aware digital image environment, include: a foreground feature machine learning system, implemented at least in part in hardware of at least one computing device, to receive a digital image having a foreground and to extract foreground features from the foreground of the digital image using a first neural network as part of machine learning; a background feature machine learning system, implemented at least in part in hardware of the at least one computing device, to receive a plurality of candidate digital images having a background and extract background features from the background using a second neural network as part of the machine learning; a score calculation module, implemented at least in part in hardware of the at least one computing device, to calculate a score by feature embedding based on the extracted foreground features from the digital image and the extracted background features from the plurality of candidate digital images; as well as A search module is implemented at least in part in the hardware of the at least one computing device to output search results including at least one of the candidate digital images based on the calculated score.

19. The system of claim 18 further comprising a category feature machine learning system, the category feature machine learning system being at least partially implemented in hardware of the at least one computing device to generate category features from category data including text describing the digital image, and embedding the category features as part of the background features and the foreground features.

20. The system of claim 19, wherein the category feature is a vector representation of the text formed using machine learning.

Citation Information

Patent Citations

  • Structured knowledge modeling and extraction from images

    CN106682059A

  • Image processing using histograms

    US20180082410A1