Concept decomposition for visual exploration

The method generates concept trees through vector embeddings and textual inversion to represent complex visual aspects, addressing the limitations of traditional methods by providing a robust and coherent visual content representation, enhancing exploration and categorization capabilities.

WO2024243526A1PCT designated stage expired Publication Date: 2024-11-28GOOGLE LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/US2024/031028
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-24
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Traditional methods for representing complex visual aspects in image collections struggle to capture nuanced relationships and diverse visual elements, leading to limited understanding and practical applicability in real-world scenarios.

Method used

A computer-implemented method generates concept trees for imagery using vector embeddings and textual inversion, enforcing binary reconstruction and coherency to decompose visual concepts into hierarchical structures, allowing for robust and coherent representation of visual content.

Benefits of technology

This approach enables accurate and efficient visual exploration, image synthesis, and conceptual categorization, improving the representation of visual content and its underlying structure, enhancing applications like image search and content recommendation systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2024031028_28112024_PF_FP_ABST
    Figure US2024031028_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are systems and methods to generate concept trees for imagery. The proposed techniques can utilize large vision-language models and their rich latent space to decompose a visual concept, represented as a set of images, into different visual aspects encoded in an hierarchical tree structure. Each node in the tree can represent a vector embedding in the latent space of a pretrained text-to-image model and can be implicitly separated into two child sub-concepts by optimizing two new embeddings. A set of regularizations can be used to guide the optimization of the embeddings encoded in the nodes to follow the hierarchical structure of the tree (e.g., to demonstrate both binary reconstruction and coherence).
Need to check novelty before this filing date? Find Prior Art

Description

CONCEPT DECOMPOSITION FOR VISUAL EXPLORATIONRELATED APPLICATIONS

[0001] This application claims priority to and the benefit of United States Provisional Patent Application Number 63 / 504,009, filed May 24, 2023. United States Provisional Patent Application Number 63 / 504.009 is hereby incorporated by reference in its entirety.FIELD

[0002] The present disclosure relates generally to computer vision. More particularly, the present disclosure relates to multimodal learning, which combines and integrates information from different ty pes of data, such as textual and visual data.BACKGROUND

[0003] Traditional approaches to representing the diverse visual aspects present in image collections have faced substantial difficulties. While these techniques can extract basic visual features from images, they often struggle to represent the complex and nuanced visual aspects that can exist within a set of images. This complexity arises from the fact that a single image can contain multiple objects, scenes, and semantic concepts, each with their own visual characteristics. Furthermore, images in a set can have interconnected or hierarchical relationships, adding another layer of complexity to the representation challenge.

[0004] Existing solutions typically fall short in adequately capturing and representing these complex relationships and diverse visual aspects in a coherent and structured way. This results in a limited understanding of the visual content and its underlying structure, which can greatly restrict the practical applicability of these systems in real-world scenarios.Consequently, there is a clear need for more advanced methods for representing the visual aspects of image sets in a robust and coherent manner.SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure is directed to computer- implemented method to generate concept trees for imagery. The method can be performed for one or more tree-building iterations. The method includes defining, by the computing system,two child nodes from a parent node of a concept tree. The method includes initializing, by the computing system, two vector embeddings respectively for the two child nodes, wherein the two vector embeddings respectively represent two words in a latent embedding space. The method includes performing, by the computing system, one or more rounds of textual inversion on the two vector embeddings while enforcing both binary7reconstruction and coherency on the two vector embeddings.

[0007] Another example aspect is directed to a computer-implemented method for performing intra-tree composition. The method includes receiving, by a computing system, an image. The method includes decomposing, by the computing system, the image into a concept-tree. The method includes receiving, by the computing system, an input selecting a plurality of the child nodes. The method includes generating, by the computing system, a synthetic image comprising aspects corresponding to the plurality7of selected child nodes. The method includes outputting, by the computing system, the synthetic image.

[0008] Another example aspect is directed to a computer-implemented method to perform text-based image generation. The method includes receiving, by a computing system, an image. The method includes decomposing, by the computing system, the image into a concept-tree. The method includes receiving, by the computing system, an input comprising a selected child node and an object that is not represented in the concept tree. The method includes generating, by the computing system, a synthetic image comprising a visual concept corresponding to the selected child node and the object. The method includes outputting, by the computing system, the synthetic image.

[0009] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Detailed discussion of embodiments directed to one of ordinary' skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0012] Figure 1 depicts a flow chart diagram of an example method according to example embodiments of the present disclosure.

[0013] Figure 2 depicts a flow chart diagram of an example method according to example embodiments of the present disclosure.

[0014] Figure 3 depicts a flow chart diagram of an example method according to example embodiments of the present disclosure.

[0015] Figures 4A-D depict graphical diagrams of example aspects of generating a concept tree according to example embodiments of the present disclosure.

[0016] Figure 5 A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[0017] Figure 5B depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0018] Figure 5C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0019] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.DETAILED DESCRIPTIONOverview

[0020] Generally, the present disclosure is directed to systems and methods to generate concept trees for imagery. The proposed techniques can utilize large vision-language models and their rich latent space to decompose a visual concept, represented as a set of images, into different visual aspects encoded in an hierarchical tree structure. Each node in the tree can represent a vector embedding in the latent space of a pretrained text-to-image model and can be implicitly separated into two child sub-concepts by optimizing two new embeddings. A set of regularizations can be used to guide the optimization of the embeddings encoded in the nodes to follow the hierarchical structure of the tree (e.g., to demonstrate both binary reconstruction and coherence).

[0021] More particularly, in some implementations, the proposed technique can include multiple iterations of tree-building. At each iteration of tree-building a computing system can define two child nodes from a parent node of a concept tree. These child nodes are each initialized with a vector embedding, which represent two words in a latent embedding space. Following initialization of the vector embeddings, the system conducts one or more rounds of textual inversion on these vector embeddings, enforcing constraints such as binary reconstruction and coherency on the two vector embeddings.

[0022] In particular, according to an aspect of the present disclosure, the execution of these textual inversion rounds can include the computing system creating multiple pairs of candidate values for the vector embeddings. For example, the multiple pairs of candidate values can be created in a way that enforces binary reconstruction, as will be described in further detail herein. Binary7reconstruction refers to the concept that the combination of the vector embeddings should encapsulate the concept depicted by their parent node.

[0023] Once multiple pairs of candidate values have been generated, the computing system can then employ an image generation model, such as, for example, a latent diffusion model, to produce a set of synthetic images for each candidate value. For example, the image generation model can be conditioned on a particular candidate value when generating synthetic image(s) for that candidate value.

[0024] The computing system can then evaluate a coherence score for each pair of candidate values. In some implementations, the coherence score for each pair of candidate values can depend partly on an inter-node consistency score that measures the consistency between the two sets of synthetic images created for each pair of candidate values. Additionally or alternatively, the coherence score for each pair of candidate values can depend partly on an intra-node consistency score measured for each candidate value. The intra-node consistency score can measure image consistency within the set of synthetic images generated for that candidate value. The computing system can then choose one pair of candidate values for the vector embeddings based on the computed coherence scores.

[0025] In particular, the inter-node consistency score can specifically evaluate the similarity between image embeddings created from the two sets of synthetic images made for each pair of candidate values. This similarity7can be calculated using a variety7of metrics, such as, for example, cosine similarity. The embeddings can be generated using various embedding generation techniques, such as, for example, use of a pretrained CLIP image encoder. In general, the computing system prefers to select a pair of candidate values that demonstrate a low inter-node consistency score. By selecting a pair of candidate values that demonstrates a low inter-node consistency score, the coherency of the two child nodes can be enhanced. Coherency refers to the concept that each individual node should depict a coherent concept which is distinct from its sibling.

[0026] In some implementations, the coherence score for each pair of candidate values additionally or alternatively takes into account two intra-node consistency scores, each of which measures consistency within the sets of synthetic images created for each candidate value. This metric further enhances the system's ability to select candidate values that eachrepresent a consistent visual concept. In general, the computing system prefers to select a pair of candidate values that demonstrate high intra-node consistency scores. By selecting a pair of candidate values that demonstrates high intra-node consistency scores, the consistency of the two child nodes can be enhanced.

[0027] In some implementations, the computing system generates the multiple pairs of candidate values in a way that enforces binary reconstruction. In particular, in some implementations, to generate the multiple pairs of candidate values, the computing system can initialize each of multiple pairs of candidate values for the vector embeddings. The computing system can then execute one or more binary reconstruction optimization iterations for each pair.

[0028] Each binary reconstruction optimization iteration can include the computing system selecting an image linked with the parent node and encoding this image into a noised latent image with added noise (e.g., via interaction with an image generation model such as a latent diffusion model). The computing system can then further interact with the image generation model to predict noise from this noised latent image while conditioning the image generation model on a conditioning text that contains both words associated with the child nodes. Thus, the computing system can use the conditioning text to prompt the image generation model to generate synthetic imagery7that visually represents a combination of the two ords.

[0029] The computing system can then update the candidate values based on a loss function that evaluates the added and predicted noise. For example, the loss function can evaluate whether the imagery generated from the combination of the tw o words represents the same concept as the original selected image from the parent node. In this fashion, the ability of the combination of the two words to encapsulate the concept depicted by their parent node ('‘binary reconstruction’’) can be measured. The computing system can update the candidate values for the embeddings of the tw o w ords in a direction that increases their ability7to encapsulate the concept depicted by their parent node, thereby increasing binary7reconstruction.

[0030] Thus, the proposed solution offers a practical and scalable approach to automatically generating concept trees for imagery using machine learning methods. This technique can be integrated into several practical applications to solve technical problems related to image and text understanding, image synthesis, and conceptual categorization. Once the tree has been generated, the computing system can visualize the concept tree. This visualization can include the parent node, child nodes, and the images associated with thechild nodes generated by the image generation model. This adds a dimension of user- friendliness to the solution, as users can view and understand the underlying structure of the concept tree.

[0031] One potential application of the proposed technique is the development of a visual exploration engine. This solution can be used to create a hierarchical structure of concepts, which can be represented as a tree with images attached to each node. A user searching for a particular type of image can navigate the tree to find exactly what they're looking for. This improves the accuracy and efficiency of visual searches, addressing the technical challenge of effectively associating images with appropriate text-based concepts.

[0032] Another example application of the proposed technique is to enable intra-tree composition. Each concept tree can be represented by a set of optimized vectors. Once this set of vectors is established, it provides the means to further explore and perform conceptual modifications within the object's conceptual domain. This can be achieved by composing sentences that encompass different subsets of the vectors corresponding to the original concept. For instance, one can create variations of the original object by combining a first vector from the set and a second vector from the set, which may result in a variation that excludes certain aspects of the original concept. Another possibility includes excluding a specific sub-concept by omitting the corresponding vector. By exploring the space of possible combinations, one can generate new perspectives on the concept and discover inspiration by focusing on specific aspects of the original concept. This method can be particularly useful in the realms of design and conceptual editing.

[0033] Another example application of the proposed technique is to enable inter-tree composition, allowing for the combination of concepts learned across different concept trees. This approach involves injecting new words into the existing dictionary’, avoiding the finetuning of model's weights as seen in other personalization approaches. To facilitate this, concept trees are initially built independently. Sub-concepts represented in the nodes of these trees are then visualized to identify interesting combinations. By placing placeholder words together in a sentence corresponding to the desired combination and feeding it into a pretrained text-to-image model, combined concepts can be generated. This methodology opens up a myriad of creative possibilities by facilitating the fusion of diverse concepts.

[0034] Another example application of the proposed technique is to enable text-based generation. For example, the place-holder words of the learned embeddings can be composed into natural language sentences to generate various scenes based on the concept used. This enables a user to explore the learned aspects as part of a potential design or idea.

[0035] The systems and methods of the present disclosure also address technical problems associated with enforcing coherency and binary reconstruction in tree nodes and corresponding generated synthetic images. By optimizing coherence scores based on internode and intra-node consistency, the solution ensures that children of the same node are able to together encapsulate the concept depicted by their parent node; while also enforcing that each node represents a visual concept sufficiently different (e g., coherent) from the other child nodes. Consistency of representation and imagery within each individual node can also be increased. Enforcing these constraints are valuable in applications like image search or exploration engines and / or content recommendation systems, where the system needs to differentiate between similar but distinct concepts.

[0036] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.Example Preliminaries

[0037] Latent Diffusion Models are a type of generative model. Their role is to leam a data distribution by gradually denoising a variable sampled from a Gaussian distribution. One example latent diffusion model is the Stable Diffusion model (Rombach et al., 2022), a publicly accessible text-to-image variant of Latent Diffusion Models. This model utilizes a pretrained image autoencoder that includes an encoder 'E' (which maps input images x into latent images z) and a decoder 'D' (which aims to reconstruct the input image x from the latent vector z, such that D(z) = x). A second-stage denoising diffusion probabilistic model (DDPM) can then be trained to generate codes within this latent space. At each step during training, a scalar t e {1, 2, .. . T is uniformly sampled and used to define a noised latent code zt= atz + crte, where e ~ N (0, 7) and atl<jtare terms that control the noise schedule, and are functions of the diffusion process time t.

[0038] The denoising network eeis fed the noised code zt. the timestep, and an optional conditioning vector c(y). Its goal is to predict the added noise. An example LDM loss can be defined as:

[0039] For text-to-image generation, the condition is based on text, where y is the given sentence in natural language and c(y) is a vector embedding encoding the text. At inference, a random latent code is sampled and iteratively denoised to generate a clean latent code, which the decoder then translates into an image.

[0040] In the Text Embedding stage, given a text prompt y (e.g., "A photo of a cat"), the sentence is tokenized and each token is mapped to a unique vector in a predefined dictionary of embeddings. These vectors are then processed by a text encoder, which processes the connections between the words in the sentence and generates an output c(y). The output c(y) can be used as a condition for the image denoising process. Words can be denoted with S and the vector embeddings from the lookup table can be denoted with V.

[0041] Textual Inversion is based on the framework proposed by Gal et al. (2022). They choose the embedding space of V as the target for inversion. They formulate the task of inversion as fitting a new word s * to represent a personal concept, depicted by a small set of input images provided by the user. They extend the predefined lookup table with a new embedding vector v * that is linked to s *. v * is initialized often with the embedding of an existing word from the dictionary, that has some relation to the given concept and then optimized to represent the desired personal concept. This process can be thought of “injecting” the new concept into the vocabulary.

[0042] v * is optimized with respect to the LDM loss in Equation (1) over images sampled from the input set. At each step of optimization, a random image x is sampled from the set, along with a neutral context texts y, (e.g., "A photo of s *"). Then, image x is encoded to z = E(%) and noised with respect to a randomly sampled timestep t and noise e zt= atz + <jte. The noised latent image zt, timestep t, and text embeddings c(y) are then fed into a pretrained decoder model which is trained to predict the noise c applied with respect to the conditioned text and timestep. This way, v * is optimized to describe the object depicted in the small training set of images.Example Methods

[0043] Given a small set of input images 7° = {if ... / ,„} depicting the desired visual concept, systems and methods of the present disclosure can construct a rich visual exploration space expressing different aspects of the input concept.

[0044] Example techniques can model the exploration space as a binary tree whose nodes V = {v±. . vn} are learned vector embeddings corresponding to newly discovered words S = {s, . . sn] added to the dictionary, representing different aspects of the original concept. These newly learned words can be used as input to a pretrained text-to-image model to generate a rich variety of image examples in each node.

[0045] A tree provides an intuitive structure to separate and navigate the different aspects of a given concept. Each level allows a user to find more aspects of the concepts in the previous level. In addition, each node by itself contains a plethora of samples from the sub-concept and can be used for exploration.

[0046] A computing system can build the tree can built gradually as a binary' tree from top to bottom, where the computing system iteratively adds two new nodes at a time. To create two child nodes, the computing system can optimize new embedding vectors according to the input image-set generated from the concept depicted in the parent node. As an example, Figures 4A and 4B show an example of this process. For example, Figure 4A shows creation of two child nodes vtand vrfor a parent node vp. Figure 4B shows the extension of an embedding table to create two new embeddings for the two child nodes vtand vr.

[0047] According to an aspect of the present disclosure, during construction, two requirements can be enforced to encourage the learned embeddings to follow the tree structure: (1) Binary Reconstruction - each pair of children nodes together should encapsulate the concept depicted by their parent node, and (2) Coherency - each individual node should depict a coherent concept which is distinct from its sibling.

[0048] As one example, Figure 1 depicts a flow chart diagram of an example method to generate a concept tree for imagery according to example embodiments of the present disclosure. Although Figure 1 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method shown in Figure 1 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure. The steps of the method shown in Figure 1 can optionally be repeated over a number of tree-building iterations, as indicated by the dashed line.

[0049] At 12, the computing system defines two child nodes from a parent node of a concept tree. For example, at a first iteration of the method shown in Figure 1, the computing system can receive a set of one or more input images (e.g., provided or otherwise selected by a user). The computing system can establish a root node associated with these input images. The computing system, at 12, can then define two child nodes from this parent node. At subsequent iterations of the method shown in Figure 1, the computing system can define two child nodes from any existing node of the concept tree, which then becomes the "‘parent” node to the two newly created child nodes.

[0050] At 14, the computing system initializes two vector embeddings respectively for the two child nodes. These vector embeddings can be defined in a latent embedding space, an abstract high-dimensional space, where they represent two distinct words. The words can (ultimately) be descriptive of the visual sub-concepts that the respective child nodes represent. The initialization of these embeddings involves the initialization of values within the latent space that correspond to the semantic meaning of these words. The generated embeddings serve as the initial state for subsequent operations, which involve rounds of textual inversion to enforce binary reconstruction and coherency.

[0051] As an example, in each optimization phase, the computing system can leam two vector embeddings and vrcorresponding to the left and right sibling nodes, whose parent node is marked with vp. At 14, the computing system can extend the cunent dictionary by adding two new vector embeddings vLand vr, corresponding to the right and left children of their parent node vp. In some implementations, to be as general as possible, the newly added vectors can be initialized with the embedding of the word “objecf ’. See, e.g.. Figures 4A and 4B as introduced above.

[0052] Referring still to Figure 1, at 16, the computing system performs one or more rounds of textual inversion on the two vector embeddings. This is done while enforcing both binary reconstruction and coherency on the two vector embeddings. Binary reconstruction ensures that the combined representations of the child nodes effectively encapsulate the concept represented by their parent node. In essence, it validates that the parent node's concept can be reconstructed by combining the sub-concepts represented by its child nodes. Coherency, on the other hand, ensures that each child node maintains a unique and coherent representation of its corresponding visual sub-concept, distinct from its sibling node. This textural inversion performed at 16 refines the vector embeddings to adhere to these constraints, thereby optimizing the quality and relevance of the visual sub-concepts represented in the concept tree.

[0053] As one example of a technique that can be performed at 16, Figure 2 depicts a flow chart diagram of an example method to perform textual inversion while enforcing coherency according to example embodiments of the present disclosure. Although Figure 2 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method shown in Figure 2 can be omitted, rearranged,combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0054] At 202, the computing system initiates the process by generating multiple pairs of candidate values for the two vector embeddings. This step involves the computation of numerous potential values which represent the two words in a latent embedding space, correlating to the two child nodes of a concept tree.

[0055] As one example of a technique that can be performed to generate candidate pairs at 202, Figure 3 depicts a flow chart diagram of an example method to generate multiple pairs of candidate values while enforcing binary reconstruction according to example embodiments of the present disclosure. Although Figure 3 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method shown in Figure 3 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0056] At 302, the computing system can initialize each of a plurality of pairs of candidate values for the two vector embeddings. This step can include preparing multiple pairs of candidate starting values that represent two words in the latent embedding space, corresponding to the two child nodes of a concept tree.

[0057] Steps 304-310 can then be performed (e.g., iteratively) for each of the plurality of pairs of candidate values.

[0058] Specifically, at 304, the computing system selects an image associated with the parent node of the concept tree. The selected image serves as the basis for subsequent steps involving encoding and noise generation.

[0059] In some implementations, at the root of the concept tree, the computing system can use the original set of images 1° for the optimization. However, in each internal node, the computing system can use the previously learned vector vpfor the parent node to generate a new small training set of images lp= { / . . . Z0, by using a pretrained text-to-image model. These images reflect the visual sub-concept depicted by the vector vpin the parent node. An example of this image generation is shown at Figure 4C. Referring again to Figure 3. at 304 one of the images associated with (e.g., generated from) the previously learned vector vpfor the parent node is selected.

[0060] Following the image selection at 304, then, at 306, the computing system encodes the selected image into a noised latent image by adding a set of added noise. The encodingprocess transforms the selected image into a noised latent image. For example, the computing system can sample an image x from the set Ipand encode the image to form the latent image z = E x). A timestep t and a noise e can also sampled to define the noised latent zt= atz + ate.

[0061] Next, at 308. the computing system utilizes an image generation model to generate predicted noise from the noised latent image. This generation process is conditioned on a conditioning text that contains both of the two words associated with the two child nodes. The output of this step is a predicted noise that corresponds to the noised latent image.

[0062] As an example, the computing system can sample a neutral context or conditioning text y. In one example, the conditioning text can contain the new placeholder words in the following form '‘A photograph of s(srNext, the noised latent ztis fed to a pretrained image generation model (e.g., a Stable Diffusion UNet model) ee, conditioned on the text embedding c(y) of the sampled text, to predict the noise e. For example, the text embedding c(y) can be generated using a pretrained text embedding model such as CLIP.

[0063] An illustration of an example implementation of steps 306 and 308 is show n in Figure 4D.

[0064] Referring again to Figure 3, at 310, the computing system updates the pair of candidate values for the two vector embeddings. This update is based on a loss function that evaluates the added noise and the predicted noise. The objective of this step is to minimize the discrepancy between the added noise and the predicted noise, thus refining the candidate values for the two vector embeddings.

[0065] As an example, the prediction loss can be backpropagated with respect to the vector embeddings v vras follows:

[0066] This procedure encourages vhvrtogether to express the visual concept of their parent depicted in the set lp.

[0067] Some example, implementations can use the timestep sampling approach proposed in ReVersion (Huang et al. 2023), which skews the sampling distribution so that a larger t is assigned a higher probability, according to the following importance sampling function:

[0068] In some examples, a = 0.5. This sampling approach helps to improve stability and content separation.

[0069] Thus, by performance of the method illustrated in Figure 3, multiple pairs of candidate values can be generated, with each pair of candidate values satisfying a binary reconstruction constraint. For example, performing the method of Figure 3 with k different seeds in parallel for a certain number of steps (e.g., 200) can results in an initial set of k pairs of candidate vector embedding values Vs= {vtl, V i=1.

[0070] Referring again to Figure 2, once the candidate values have been generated at 202, then, at 204, the computing system leverages an image generation model to generate a set of one or more synthetic images for each of the candidate values. The generation of these synthetic images is conditioned on each individual candidate value, effectively producing a unique set of synthetic images that correspond to each candidate value. For example, for each vector v S fy. the computing system can generate a random set Ivof images (e g., of magnitude 40) using a pretrained image generation model.

[0071] At 206, the computing system determines a coherence score for each pair of candidate values. The coherence score can be based on an inter-node consistency score that measures the consistency between the two sets of synthetic images generated for the pair of candidate values. For example, the inter-node consistency score can provide a quantitative measure of how closely associated or consistent the synthetic images are for each pair of candidate values. The coherence score can also be based on an intra-node consistency score that measures consistency within the set of images generated for each node.

[0072] At 208, the computing system then proceeds to select one pair of candidate values for the two vector embeddings. The selection process is predicated on the coherence scores, where the pair with the highest coherence score (e.g., indicating high consistency within each set of synthetic images and low consistency between sets of images) is selected as the optimal pair of vector embeddings.

[0073] As an example, in some implementations, the computing system can use a pretrained image encoder (e.g., the CLIP image encoder), to produce the embedding of each image in each set, with each embedding expressed, for example, as follows: CLIP (If).

[0074] The consistency of two sets of images Ibcan be measured as follows:

[0075] Note that I C(la, / b) | < 1 because sim(x, y) = ||x||-y||y|| is the cosine similarity between a pair of CLIP embeddings of two different images. This formulation is motivatedby the observation that if a set of images depicts a certain semantic concept, their vector embedding in CLIP’S latent space should be relatively close to each other.

[0076] Ideally, the computing system will select pairs in which each node is coherent by itself, and in addition, two sibling nodes are distinct from each other. In some implementations, therefore, the computing system can select the pair of tokens {v ,as follows:

[0077] Note that the above expression not only takes the absolute inter-node consistency score C into consideration, but it is computed with respect to the minimum intranode consistency.

[0078] In some implementations, once the selection has occurred, the computing system continues the optimization of the embeddings of the selected pair with respect to the reconstruction loss in Equation (2) for some number of iterations (e.g., 1500).

[0079] Referring again to Figure 1, after or during the building of the concept tree via iterative performance of steps 12-16, the computing system can visualize the concept tree at step 18. The visualization can be a user-friendly presentation of the hierarchical structure of the concept tree, which includes various elements such as the parent node, the child nodes, and the images associated with these child nodes. These images are synthetically generated by the image generation model to visually represent the concepts encapsulated by the nodes. Visualizing the concept tree at 18 enables users to comprehend the hierarchical and interconnected relationships betw een different visual concepts represented in the concept tree.

[0080] The visualization of the concept tree can be presented through a user interface that allows a user to explore and traverse the tree in an interactive manner. For example, the user interface can present the root node at the top of the interface, with its child nodes branching out below. Each node can be associated with a representation of the synthetic image representing its concept, providing a visual cue to the user. Users can select any node to access its child nodes, effectively explonng more specific visual concepts. The user interface can also include functionality for the user to navigate back up the tree, allowing exploration of different branches.

[0081] Furthermore, additional features can be integrated into the user interface to enhance the exploration experience. For instance, a search function can allow users to locate specific nodes or concepts within the tree. Hovering over or selecting a node can provide more detailed information about the concept it represents, such as a textual description or additional related images. Moreover, the interface can offer the ability to manipulate or rearrange the tree, enabling users to customize the visualization based on their preferences or needs. Through these user interface features; users are enabled to interactively explore and understand the complex relationships between different visual concepts in the concept tree.Example Devices and Systems

[0082] Figure 5A depicts a block diagram of an example computing system 100 to generate a concept tree according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a netw ork 180.

[0083] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g.. laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0084] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality’ of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 1 16 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0085] In some implementations, the user computing device 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural netw orks can include feed-forward neural netw orks, recurrent neural netw orks (e.g., long short-term memory' recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some examplemachine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0086] In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine-learned model 120.

[0087] Additionally or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 140 as a portion of a web service (e.g., a concept tree generation and / or exploration service). Thus, one or more models 120 can be stored and implemented at the user computing device 102 and / or one or more models 140 can be stored and implemented at the serv er computing system 130.

[0088] The user computing device 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0089] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0090] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such servercomputing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0091] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0092] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the sen' er computing system 130.

[0093] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transilory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0094] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0095] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0096] In particular, the model trainer 160 can train the machine-learned models 120 and / or 140 based on a set of training data 162. In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0097] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0098] The network 180 can be any type of communications network, such as a local area network (e.g.. intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP. HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0099] The proposed system can also include a concept tree generator 170, which can be integrated into the server computing system 130, user computing device 102, or function as a standalone component coupled with the network 180. The concept tree generator 170 is specifically designed to implement the methods described herein for constructing concept trees as per the proposed solution. It can employ the machine-learned models 120 and / or 140 and / or associated learning infrastructure to generate synthetic images, calculate coherence scores, and update the candidate values for the vector embeddings. The concept tree generator 170 can function independently or collaboratively with the user computing device 102 and server computing system 130 in the generation of concept trees.

[0100] The concept tree generator 170 can include one or more processors and a memory. The processor can be any suitable processing device (e.g.. a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory’ devices, magnetic disks, etc., and combinations thereof. The memory can store data and instructions which are executed by the processor to perform operations as per the method steps in the proposed solution.

[0101] The concept tree generator 170 can interact with the training computing system 150 and receive trained machine-learned models for its operations. Furthermore, it can receive user inputs via the user computing device 102, process these inputs, and generate corresponding outputs. These outputs can be sent back to the user computing device 102 or server computing system 130 for further interaction with the user or for storage and subsequent analysis. The concept tree generator 170 can enable the creation and / or visualization of meaningful and coherent concept trees.

[0102] Figure 5 A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.

[0103] Figure 5B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0104] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0105] As illustrated in Figure 5B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using anAPI (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0106] Figure 5C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0107] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0108] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 5C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0109] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 5C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).Additional Disclosure

[0110] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single systemor distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0111] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method to generate concept trees for imagery, the method comprising, for each of one or more tree-building iterations: defining, by the computing system, two child nodes from a parent node of a concept tree; initializing, by the computing system, two vector embeddings respectively for the two child nodes, wherein the two vector embeddings respectively represent two words in a latent embedding space; and performing, by the computing system, one or more rounds of textual inversion on the two vector embeddings while enforcing both binary' reconstruction and coherency on the two vector embeddings.

2. The computer-implemented method of claim 1, wherein performing, by the computing system, the one or more rounds of textual inversion comprises: generating, by the computing system, multiple pairs of candidate values for the two vector embeddings; generating, by the computing system using an image generation model, a set of one or more synthetic images for each candidate value by conditioning the image generation model on the candidate value; determining, by the computing system, a coherence score for each pair of candidate values, wherein the coherence score for each pair of candidate values is based at least in part on an inter-node consistency7score that measures a consistency between the two sets of synthetic images generated for the pair of candidate values; and selecting, by the computing system, one of the pairs of candidate values for the two vector embeddings based on the coherence scores.

3. The computer-implemented method of claim 2, wherein the inter-node consistency score measures a similarity between image embeddings generated from the two sets of synthetic images generated for the pair of candidate values.

4. The computer-implemented method of claim 2 or 3, wherein the coherence score for each pair of candidate values is further based at least in part on two intra-node consistency scores that respectively measure consistency within each of the two sets of synthetic images generated for the pair of candidate values.

5. The computer-implemented method of claim 4. wherein each intra-node consistency score measures similarity between image embeddings generated from images within the set of synthetic images.

6. The computer-implemented method of claim 4 or 5, wherein the coherence score comprises the two intra-node consistency scores plus a minimum of the two intra-node consistency scores minus the inter-node consistency score.

7. The computer-implemented method of claim 2 or 3, wherein generating, by the computing system, the multiple pairs of candidate values for the two vector embeddings comprises: initializing, by the computing system, each of the pairs of candidate values for the two vector embeddings; and performing binary reconstruction optimization iterations for each pair of candidate values, each binary' reconstruction optimization iteration comprising: selecting, by the computing system, an image associated with the parent node; encoding, by the computing system, the selected image into a noised latent image by adding a set of added noise; generating, by the computing system using the image generation model, predicted noise from the noised latent image and conditioned on a conditioning text containing both of the two words associated with the two child nodes; and updating, by the computing system, the pair of candidate values for the two vector embeddings based on a loss function that evaluates the added noise and the predicted noise.

8. The computer-implemented method of any preceding claim, further comprising: visualizing, by the computing system, the concept tree, wherein the visualization includes the parent node, child nodes, and images associated with the child nodes generated by the image generation model.

9. A computing system configured to perform the method of any preceding claim.

10. One or more non-transitory computer-readable media that store computerexecutable instructions for performing the method of any of claims 1-8.

11. A computer-implemented method for performing intra-tree composition, the method comprising: receiving, by a computing system, an image; decomposing, by the computing system, the image into a concept-tree according to the method of claim 1 ; receiving, by the computing system, an input selecting a plurality of the child nodes; generating, by the computing system, a synthetic image comprising aspects corresponding to the plurality of selected child nodes; and outputting, by the computing system, the synthetic image.

12. A computer-implemented method to perform text-based image generation, the method comprising: receiving, by a computing system, an image; decomposing, by the computing system, the image into a concept-tree according to the method of claim 1 ; receiving, by the computing system, an input comprising a selected child node and an object that is not represented in the concept tree; generating, by the computing system, a synthetic image comprising a visual concept corresponding to the selected child node and the object; and outputting, by the computing system, the synthetic image.

Citation Information

Patent Citations

  • US202363504009P

  • US63504009P