Method, system and computer readable medium for learning latent structural relationships

Abstract

Learning dissociated representations is an important topic in machine learning in many applications. Dissociated latent variables represent interpretable semantic information and reflect the separation factors of changes in the data. Although generative models can learn latent representations and generate data samples, existing models may ignore the structural information between latent representations. Described in the present disclosure is an embodiment of learning dissociated latent structural representations from data using a decomposable variational autoencoder that simultaneously learns component representations and encodes component relationships. An embodiment of a novel structural prior for the latent representation is disclosed to capture the interactions between different data components. The embodiment is applied to data segmentation and latent relationship discovery between different data components. Experiments on several datasets demonstrate the practicality of the present model embodiment.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for computer learning that can provide improved computer performance, features, and uses. More specifically, the present disclosure relates to systems and methods for learning latent structural relationships for improved computer performance, features, and uses. Background Art

[0002] Deep neural networks have achieved great success in many fields, such as computer vision, natural language processing, and recommendation systems. Disentangled representation learning aims to learn to discover and disentangle the factorized representation of potential explanatory factors in data. Disentangled representation is a fundamental but challenging problem in machine learning and artificial intelligence. Interpretable disentangled representations have demonstrated their role in unsupervised and semi-supervised learning.

[0003] The main challenges in extracting representations from images with multiple objects lie in the unsupervised setting and complex interaction patterns. Most existing methods may not be applied to this problem because it is challenging to integrate data segmentation and representation learning. In addition, learning complex entity interactions in the real world requires strong and flexible priors for the latent representation, which can adaptively encode complex structural relationships.

[0004] Therefore, there is a need for systems and methods for learning latent structural relationships for improving computer performance, features, and usage. Summary of the invention

[0005] The first aspect of the present invention provides a method for learning potential structural relationships. The method includes receiving an input including one or more components; using an attention network, generating a mask corresponding to each of the one or more components, the mask indicating the probability that at least a subset of the elements of the input belong to each of the one or more components; using an encoder, encoding the input and each mask into a latent variable, the latent variable including a global latent variable and a local latent variable of the corresponding component; using a flow-based model including one or more flow functions, transforming the one or more global latent variables into one or more transformed global latent variables, each flow function forward transforming one of the one or more global latent variables to obtain a corresponding transformed global latent variable; generating an aggregated global latent variable based on the one or more transformed global latent variables; using a flow-based model, transforming the aggregated global latent variable into one or more reconstructed global latent variables corresponding to the one or more components; using a decoder, by using a reconstructed global latent variable and a local latent variable corresponding to each of the one or more components, generating a pair of reconstructed masks and reconstructed components for each of the one or more components; and using the reconstructed components and the reconstructed masks corresponding to the one or more components for one or more data processing applications in the inference process, or for constructing one or more losses for training in the training process.

[0006] A second aspect of the present invention provides a system for learning potential structural relationships. The system includes one or more processors and a non-transitory computer-readable medium. The non-transitory computer-readable medium includes one or more instruction sets. When executed by at least one of the one or more processors, the instructions cause at least one processor to perform the method according to the first aspect.

[0007] A third aspect of the present invention provides a non-transitory computer-readable medium, comprising one or more instruction sequences, which, when executed by at least one processor, cause the at least one processor to perform the method according to the first aspect.

[0008] A fourth aspect of the present invention provides a computer program product, comprising a computer program, wherein the computer program implements the method according to the first aspect when executed by a processor.

[0009] The method for learning latent structural relationships provided by the present invention takes into account the interactions between multiple objects or sample parts, and has achieved significant improvements in scene segmentation and object representation learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Reference will be made to embodiments of the present disclosure, examples of which may be shown in the accompanying drawings. These drawings are illustrative only and not restrictive. Although the present disclosure is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the present disclosure to these specific embodiments. The items in the drawings may not be drawn to scale.

[0011] Figure 1 An aggregate prior model for encoding and decoding according to an embodiment of the present disclosure is depicted.

[0012] Figure 2 Depicted is a hierarchical structure for latent variables according to an embodiment of the present disclosure.

[0013] Figure 3 A framework for generating a hierarchical model according to an embodiment of the present disclosure is depicted.

[0014] Figure 4 A process for encoding and decoding all components according to an embodiment of the present disclosure is depicted.

[0015] Figure 5 A process for global latent variable encoding and decoding according to an embodiment of the present disclosure is depicted.

[0016] Figure 6 A process of causality detection according to an embodiment of the present disclosure is described.

[0017] Figure 7 Depicted are original images, reconstructed images, and mask images for a multi-object network (MONet) and the resulting hierarchical model according to an embodiment of the present disclosure.

[0018] Figure 8 Depicted are dissociation scores over time for different methods for 2-object and 3-object datasets according to embodiments of the present disclosure.

[0019] Fig. 9 Depicted are segmentation scores of different methods at different epochs on an evaluation dataset according to an embodiment of the present disclosure.

[0020] Fig.10 Depicted are ARI scores over iterations on a domino image, according to an embodiment of the present disclosure.

[0021] Fig.11 Depicted are the values ​​of different terms lost in iterations according to an embodiment of the present disclosure.

[0022] Fig.12 Depicted is a simplified block diagram of a computing device / information processing system in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these details. In addition, those skilled in the art will recognize that the embodiments of the present disclosure described below can be implemented in a variety of ways, such as a process, device, system, apparatus, or method on a tangible computer-readable medium.

[0024] The components or modules shown in the figures are illustrations of exemplary embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It should also be understood that throughout the discussion, components may be described as separate functional units, which may include subunits, but those skilled in the art will recognize that various components or portions thereof may be divided into separate components or may be integrated together, including, for example, in a single system or component. It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

[0025] In addition, the connections between the components or systems in the figure are not intended to be limited to direct connections. Instead, the intermediate components can modify, reformat or otherwise change the data between these components. Similarly, more or fewer connections can be used. It should also be noted that the terms "coupled", "connected", "communication coupled", "interfaced", "interfaced" or any derivatives thereof should be understood to include direct connections, indirect connections through one or more intermediate devices, and wireless connections. It should also be noted that any communication, such as signals, responses, replies, confirmations, messages, inquiries, etc., can include one or more information exchanges.

[0026] References in the specification to "one or more embodiments", "preferred embodiments", "one embodiment", "multiple embodiments", etc. mean that the specific features, structures, characteristics, or functions described in conjunction with the embodiments are included in at least one embodiment of the present disclosure, and may be in more than one embodiment. Likewise, the appearance of the above phrases in various places in the specification does not necessarily all refer to the same embodiment or embodiments.

[0027] The use of certain terms in various places in the specification is for illustrative purposes and should not be construed as limiting. Services, functions, or resources are not limited to a single service, function, or resource; the use of these terms may refer to groupings of related services, functions, or resources, which may be distributed or aggregated. The terms "include" and "comprise" should be understood as open terms, and any list below is an example and is not meant to be limited to the items listed. A "layer" may include one or more operations. The words "optimal," "optimized," and the like refer to improvements in results or processes, without the need for a particular result or process to have reached an "optimal" state or peak state. The use of memory, database, repository, data store, table, hardware, cache, and the like may be used herein to refer to one or more system components in which information may be entered or otherwise recorded.

[0028] In one or more embodiments, stopping conditions may include: (1) a set number of iterations have been executed; (2) a certain processing time has been reached; (3) convergence (e.g., the difference between consecutive iterations is less than a first threshold); (4) divergence (e.g., performance degradation); and (5) acceptable results have been achieved.

[0029] Those skilled in the art will recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order listed herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0030] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims.Each reference / document mentioned in the patent document is incorporated herein by reference in its entirety.

[0031] It should be noted that any experiments and results provided herein are provided in an illustrative manner and are performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results shall not be used to limit the scope of disclosure of the current patent document.

[0032] It should also be noted that although the embodiments described herein may be within the context of image processing, aspects of the present disclosure are not limited thereto. Therefore, aspects of the present disclosure may be applied or adapted for use in other contexts, including but not limited to language processing.

[0033] A. General Introduction

[0034] Deep neural networks have achieved great success in many fields, such as computer vision, natural language processing, and recommendation systems. Dissociated representation learning aims to learn to discover and dissociate the decomposed representation of potential explanatory factors in data. Dissociated representation is a fundamental but challenging problem in machine learning and artificial intelligence. Interpretable dissociated representations have demonstrated their role in unsupervised and semi-supervised learning.

[0035] Most existing methods for dissociated representation learning are based on variational autoencoders (VAEs) or generative adversarial networks (GANs). What these works have in common is that the dissociated representations are extracted from a single entity or object in a data sample. Recently, there has been a growing interest in integrating representation learning with scene segmentation using generative models. Similarly, few methods consider the interactions between multiple objects or sample parts.

[0036] The main challenges in extracting representations from images with multiple objects lie in the unsupervised setting and complex interaction patterns. Most existing methods may not be applied to this problem because it is challenging to integrate data segmentation and representation learning. In addition, learning complex entity interactions in the real world requires strong and flexible priors for the latent representation, which can adaptively encode complex structural relationships.

[0037] In the present disclosure, embodiments of novel methods for learning object representations and encoding object relationships are presented. In one or more embodiments, the latent representation vector of each object or component in a scene is divided into two parts, a local part and a global part. First, the local part controls a single characteristic that is independent of other objects. The global part shared by all objects in the scene encodes object relationships as well as global latent factors. In one or more embodiments, flow-based models can be used to handle reasoning and interactions between different objects. The flow-based structural prior of the latent representation allows rigorous score calculations to estimate correlations and causal interactions between two components.

[0038] Embodiments of the present model have been applied to different datasets, and significant improvements have been achieved in scene segmentation and object representation learning by considering the interactions between different components. Embodiments capture significantly more relationships between objects than existing methods. Theoretical properties of two-level variational autoencoder embodiments, such as the Evidence Lower Bound (ELBO), are also provided.

[0039] B. Some related work

[0040] Embodiments of the present disclosure are used to develop a method for disentangling structural latent representations using deep generative models, which has direct applications in computer vision, image processing, and many other fields that also need to solve data segmentation or data decomposition. In this section, some related work on scene or image segmentation in computer vision and disentangled learning will be reviewed.

[0041] 1. Scene segmentation

[0042] Recently, deep generative models have been integrated with unsupervised scene segmentation methods. Some people have proposed a method to simultaneously learn the representation of a single object and scene segmentation. This method of integrating iterative amortized inference and VAE is a completely unsupervised way to learn visual concepts. Using this method, the entire system can be trained end-to-end by simply maximizing its ELBO. MONet uses a recurrent attention network to distinguish different objects instead of using complex amortized inference. The scene is segmented using targets weighted by attention masks. At least one of the main differences between the embodiments of the present disclosure and the aforementioned methods is that objects in the scene can interact with each other in the present disclosure without independence assumptions between them.

[0043] 2. Dissociation

[0044] Variants of VAE have achieved state-of-the-art (SOTA) performance for unsupervised disentangled learning. A specific prior P(z) on the latent space can be assumed, and then the conditional probability P(x|z) is parameterized using a deep neural network. The distribution P(z|x) is approximated using the variational distribution Q(z|x). The objective function of VAE can be expressed as:

[0045]

[0046] The objective function is the ELBO. It is also possible to introduce various properties of the final representation by modifying the KL term. A β-VAE has been proposed where a hyperparameter β is introduced for the KL regularization term of the naive VAE. When β>1, the β-VAE penalizes the mutual information between the latent representation and the data samples. There are several different approaches to learning disentangled data representations. Independent component analysis (ICA) has been extended to the nonlinear case to achieve disentanglement of variables.

[0047] Embodiments of the present disclosure are directed to developing a framework that can seamlessly integrate data segmentation and representation learning. In this patent disclosure, a prior and theoretical analysis of latent relation learning with a message passing scheme is disclosed; an embodiment of a two-stage VAE framework with reliable derivation of ELBO is proposed; an embodiment of the proposed framework has been applied to latent relation representation learning and component segmentation. Experiments show that the segmentation and dissociated representation of different components can be improved through the reasoning mechanism of the proposed structured VAE using new prior techniques.

[0048] C. Example of Decomposing Latent Variables

[0049] In this section, we first introduce an embodiment of information aggregation priors. After that, we propose some detailed framework embodiments for learning dissociative structured latent factors. In one or more embodiments, components can refer to different parts of the current object in the image or in the data sample.

[0050] In the present disclosure, embodiments of aggregate models for learning interactions between data components are disclosed. The aggregate prior is then extended to a two-stage decomposable variational autoencoder (VAE) that can learn dissociated latent structural representations from input data. Unlike some previous methods that ignore component or object interactions, embodiments of the present disclosure simultaneously learn component representations and encode component relationships using a two-stage VAE structure. In one or more embodiments, the second level or second layer of autoencoders are parameterized using a flow-based model that allows relational reasoning to be performed, and can also be used as a structural prior for part of the latent distribution of the first layer of autoencoders. In the present disclosure, theoretical property proofs, some empirical results, and detailed network architecture embodiments are provided. The symbols in one or more model embodiments of the two-stage decomposable VAE are listed in Table 1 below.

[0051] Table 1: Some symbols used in one or more exemplary embodiments.

[0052]

[0053]

[0054] 1. Example of Latent Relation Learning with Message Passing Prior

[0055] Figure 1 An aggregate prior model according to an embodiment of the present disclosure is depicted. Figure 1 As shown in K ] is the observed data 110, y k120 comes from data component k, and h 130 is a latent variable. Given an observation y, the latent variable h can be inferred by forward message passing (encoding) 140, and the reconstruction of y can be obtained by backward message passing (decoding) 145. Here, h k =f k (y k ), and reconstruction

[0056] In one or more embodiments, y k The relationship between (k=1, ..., K) and h can be modeled using a reversible flow-based network. In one or more embodiments, the flow function f k Specify from y k The distribution of the latent variable h k The parameterized reversible transformation of f k : It can be reversible. Where l is h k and k The dimension of h k =f k (y k ), using the change-of-variables formula, we can get the following equation:

[0057]

[0058] like Figure 1 As shown in k (k=1,…,K) as the encoding (f=[f1,f2,…,f K ]) and decoding In one or more embodiments, h is obtained by aggregating all f k The output encoding y is, Therefore, the aggregated latent variable h can be a concise representation that can fully reconstruct all components of the data. In one or more embodiments, it can be enforced that f k (k=1,...,K) to ensure h k =h and

[0059] Latent variable aggregation In one or more embodiments, assuming h k Each item of (k=1,...,K) follows a normal distribution, that is, h k ~N(μ k ,σ 2 ). In one or more embodiments, the variance σ 2is set to a fixed value across all components. When the prior distribution of each item of h is normal distribution N(μ,σ 2 ). Based on the encoder and decoder VAE scheme, the model parameters of the aggregation model can be learned by maximizing ELBO.

[0060]

[0061] Given a batch of training samples, the ELBO value can be calculated using a message passing process. In one or more embodiments, Used by q f (h|y) generates samples. Given h, it is expected that it can completely reconstruct the input data. In one or more embodiments, the reconstruction term in the ELBO of equation (1) It can be calculated using the following equation:

[0062]

[0063] here In one or more embodiments, using and σ 2 Both are constant values, so the value of C can also be set to a constant. In one or more embodiments, h from a batch of training samples is used to approximate the KL term in (2).

[0064] In one or more embodiments, given a data sample y=[y1, y2, ..., y K ], you can get information about Figure 1 The following lemma on the likelihood values ​​computed using the message passing scheme.

[0065] Lemma 1. The log-likelihood of y can be approximated by the following equation

[0066]

[0067] here

[0068] Proof: h and y = [y1, y2, ..., y K The structural relationship between Figure 1 Given in The mapping from h to reconstruction is defined in Using the training objectives in equations (2) and (3), we can get The variation of the law of variables is known in the context of geometric measure theory as the smooth coarea formula, which can be described as:

[0069]

[0070] From equation (5), we can obtain the log-likelihood of y.

[0071] In one or more embodiments, each stream function f may be calculated k The Jacobian matrix of is obtained to obtain the log-likelihood value. In one or more embodiments, the correlation or causal relationship between two data components is estimated. In an example with two components a and b, in order to estimate the value of component b using the observed value of a, first estimate the value of The potential value h of . Then, we can get the estimated for At the same time, the conditional probability can be written as:

[0072]

[0073] In one or more embodiments, an inference scheme may be applied to component relationship detection, which is discussed in more detail in the section on component interaction inference.

[0074] Identifiability k,i As y k In one or more embodiments, if there is a mapping or function link y u,i and v,j , then define y u,i and v,j A relationship r between . A relationship set r may include multiple relationships. In one or more embodiments, r is used to represent a set of variables involved in r. In one or more embodiments, a relationship set in the present disclosure may be a connected graph having r as a vertex set. For the dataset For the set of all relation sets in , it is easy to prove the following lemma about relation recovery.

[0075] Lemma 2. If The relationship between the variables in is monotonic, and but Almost complete recovery is possible.

[0076] Proof of Lemma 2: The ELBO of the aggregation model is

[0077]

[0078] here

[0079] Take N(u,σ 2 ) as the prior for each term of h, we can use Approximate KL term.

[0080] here yes The i-th item of The KL term regularizes the distribution of all terms of h so that they are individually close to the prior N(u,σ 2 ), and are therefore independent of each other.

[0081] Without general loss, we can assume that σ y =σ=1. Maximizing ELBO may be equivalent to the following optimization problem,

[0082]

[0083] st

[0084] f k (y k )≥0,k=1,…,K

[0085] In one or more embodiments, a sigmoid function may be used as each f k The last step is to set u to a non-negative function. In the example of a simple case with two variables y1 and y2 and l = dim(h) = 1, assuming the relationship y1 = φ(y2) and φ is continuous, monotonic and reversible, the objective equation (7) can be rewritten as:

[0086]

[0087] st

[0088] f1(y1)≥0,f2(y2)≥0

[0089] In one or more embodiments, it can be obtained that f1(y1)=f2(y2)=h, where y2=φ(y1), therefore, and For multiple variables, let is one of the sets of relations involving multiple variables belonging to v. Each set of relations corresponds to a variable in v with variables from different components y k The interaction graph of vertex variables (k=1,...,K). Under the assumption that the relations in are monotonic and reversible, any pair of variables from v can be linked to the function. Assuming that any pair of variables from different sets of relations or graphs are independent of each other, the VAE model can transform the independent latent variables h into i One of is assigned to each relation set or graph. Therefore, as long as We can use the 25 independences of the latent variable h to approximately fully recover The set of relations in .

[0090] Using a reversible flow-based model, the disclosed two-stage decomposable VAE embodiment can be adapted to a nonlinear ICA framework. For component k, assume that k The distribution of is a factorial member of the exponential family with m sufficient statistics, with u k Here u k is an additional observed variable. In one or more embodiments, the general form of the distribution can be written as:

[0091]

[0092] Here Q i is the basic metric, Z i is the normalization constant, T i,j is a sufficient statistic component, and λ i,j is the corresponding parameter, which depends on u k The variable y k is the output of an arbitrarily complex, unavoidable, and deterministic transformation from latent space to data space, i.e. Let T = [T1, ..., T l ],λ=[λ1,...,λ l ]and The parameters The following equation can be obtained:

[0093]

[0094] In one or more embodiments, a learning algorithm may be used, namely, Get parameters A collection of . Use g k express The learned approximation of y k =g k (h k ), we can define the following equivalence relation about Θ.

[0095] Definition 1 Let ~ be about If the following equation is performed, equation (10) is recognizable to reach ~

[0096]

[0097] The elements of the quotient space Θ / ~ are called 180 identifiability classes.

[0098] Definition 2 Let ~ be about The binary relationship is defined as:

[0099]

[0100] where A is a lm×lm matrix and c is a vector of size lm. If A is reversible, the above relationship is expressed by ~ A express.

[0101] In one or more embodiments, the explicit additional observable variable u of component k k may not be available. However, K-1 signals from other components related to it may also be available. Assuming that the relationship involving component k can be fully recovered and sufficient label support can be obtained from other components, the model is recognizable. In one or more embodiments, y -k Used to represent components other than component k, and u k (y -k ) are additional variables recovered from their relations with the other components. In the limit of infinite data and good convergence, the estimated model can give the same conditional likelihood for all data points as the true generative model:

[0102]

[0103] In one or more embodiments, The domain is defined as The following theorem about the identifiability of the model can be obtained.

[0104] Theorem 1. Suppose that the data observed are distributed according to the generative model given by equations (7) and (8) and follow the following assumptions,

[0105] (a) Sufficient Statistics T ij (h) is differentiable almost everywhere, and for all And for all 1≤i≤l and 1≤j≤m, their derivatives is almost certainly non-zero;

[0106] (b) The relationship involving component k can be almost completely recovered and can be expressed as u k (y -k ) means; and

[0107] (c) From y -k There are lm+1 different conditions Make a matrix of size lm×lm It is reversible;

[0108] Then, the model parameters Yes~ A Recognizable.

[0109] The proof and analysis of Theorem 1 are as follows. Real-world datasets with non-stationary component locations are usually more complex. This patent disclosure discloses an embodiment of a two-level latent model that integrates an aggregate prior model, an attention mechanism, and component segmentation to improve flexibility.

[0110] Proof of Theorem 1: Assume and The conditional probability of is the same in the limit of infinite data. By expanding both pdfs using the variable substitution rule, we obtain the following equation:

[0111]

[0112] Unlike the approach of using observed auxiliary variables as conditioning variables, it is assumed that the relationship with component k can be recovered and the signals from other components can be used as conditioning markers. Subtract from this expression To obtain a condition and removed the Jacobian terms since they do not depend on u k , we can get the following equation:

[0113]

[0114] In equation (11), given u k h k Both conditional distributions of belong to the exponential family. Equation (11) can be rewritten as:

[0115]

[0116] Here, the basic measure Q i are canceled because they do not depend on u k . Equation (12) can be rewritten using the inner product as:

[0117]

[0118] When the lm equations are combined, equation (15) can be rewritten in matrix equation form as follows:

[0119]

[0120] here By adding L T The inverse of is multiplied by the two magnitudes of equation (16), and the following equation can be obtained:

[0121]

[0122] here, and c=L -1T b. There may be m different values to For all 1≤i≤l, such that exist are linearly independent. By defining m vectors from multiple points The Jacobian can be obtained where each term is a Jacobian of size lm×l, from the derivative of equation (17) with respect to these m vectors. Therefore, Q is lm×lm invertible, and the fact is that each component of T is univariate. In one or more embodiments, a corresponding matrix with the Jacobian calculated at the same point can be constructed And the following equation can be obtained:

[0123]

[0124] here and A are both full rank, since Q is full rank.

[0125] 2. Example of a two-level latent structure

[0126] In this section, embodiments of a hierarchical generative model that can identify components in a dataset are disclosed. The generative model uses a latent space Manifold mapped to the input space of embedded samples Generator of . Assume that the samples used in the dataset have K conditionally independent components. x=x1...x K is the output variable of the generator, and z=z0z1...z K is the latent variable of the generator, where x k is the variable of the kth component, z0 controls the global characteristics of each sample of x across all components, and z k Controls the properties of component k independently of other components and z0. In one or more embodiments, assume that if i≠k, then given the latent variable x i ⊥x k In one or more embodiments, if i≠k, it is also assumed that the components and the latent variable x are independent. i ⊥z k |z0. Based on these two assumptions, the distribution of the generated samples can be shown as:

[0127]

[0128] In one or more embodiments, a hierarchical structure is employed for latent variables. Figure 2The structure for latent variables according to an embodiment of the present disclosure is depicted. For component k, the corresponding latent variable is z′ k =z k z 0(k) .like Figure 2 As shown in the left part of , each z is a priori 0(k) 204 via the reversible function f k 230 is linked to the global latent variable z0 202. Figure 2 As shown in the right part of , z0 202 is a shared latent variable of K components 210. The deterministic pixel-wise mean and mask are expressed as and D is the dimension of the input data sample. In one or more embodiments, it can be based on 214 (decoded from the global latent variable z0 202) and z k 206 Generate reconstructed components 222 and reconstructed mask 224. Based on the reconstructed components 222 and reconstructed mask 224 generates output 226.

[0129] like Figure 2 As shown in K is the first layer of latent representation, where each z′ k =z k z 0(k) , z0 is the second layer potential representation. In one or more embodiments, z0 encodes the global characteristics of the generated samples and the correlation or interaction between different components. In one or more embodiments, z 0(k) 204 is used to represent the global information about component k decoded from z0. 0(k) and z k , z′ k Contains all the latent information of component k. In the illustrative example, a human face is used with different components representing different parts of the face, such as eyes, hair, facial skin, mouth, etc. The common latent factor z0 includes factors such as age or emotion that control the overall appearance of the face. In one or more embodiments disclosed in this patent, a framework is developed to encode individual features of each component as well as a global latent factor for the entire scene.

[0130] 3. Network Framework Implementation

[0131] Embodiments of a framework are disclosed that encode and decode each component and also capture a global latent factor. In one or more embodiments, a single VAE framework can be used for encoding and decoding of all components. For each component, the latent vector z′ k It consists of two parts, namely z′ k =z k z 0(k) .z k The local latent features for component k, and z 0(k) Features for component k controlled by the global latent factor z0.

[0132] Figure 3 The framework of the generative hierarchical model 300 according to an embodiment of the present disclosure is depicted. The generative hierarchical model 300 includes a specific network used as an attention network a for component segmentation. Figure 3 As shown in FIG. 1 , a is an attention network 310, e is an encoder 320, d is a decoder 330, and f is an k is the flow inference network 325 of component k. (x,m k ) and (z k ,z 0(k) ) are the kth component input and output of encoder 320. and is the input and output of decoder 330. In one or more embodiments, the input range of the kth component may be defined as

[0133] Figure 4 Describes an embodiment of the present disclosure for Figure 4 The process of encoding and decoding the components shown in . An input x including K (K ≥ 1) components is received (405) as an input to the model. The input can be an image with components representing objects in the image, or a data sample with components representing different parts of the data sample. In component k, the input x and the range s k As input, the attention network a outputs (410) mask m k , the mask is used to indicate the probability that at least a subset of the elements of the input x belongs to each of the one or more components. The elements of the input may be referred to as pixels of an image (when the input is an image) or data points of a data sample (when the input is a data sample). In one or more embodiments, the probability is a binary value indicating whether each pixel of x belongs to each component. Here s k is the attention residual from component 1 to k-1, that is and s1=1. In one or more embodiments, masks corresponding to one or more components are generated sequentially, for example, the range of component k can be calculated based on the range and the mask from the previous component, for example and here Represents element-wise multiplication.

[0134] Encoder e combines the input of the kth component and the mask (x,m k ) encoding (415) as the overall latent variable z′ of the kth component k (z′ k =z k z 0(k) ). k The local latent features for component k, and z 0(k) Features for component k controlled by the global latent factor z0. In one or more embodiments of the present disclosure, unlike MONet using only one layer of latent variables, one or more stream functions (f = {f1, ..., f k}) is used as the second-layer autoencoder to transform all global latent variables z 0(k) , k=1,…K is transformed into (420) the transformed global latent variable (f k (z 0(k) ), k = 1, ... K. Then, an aggregated global latent variable (z0) is generated (425) based on the one or more transformed global latent variables. The aggregated global latent variable z0 is transformed (430) back to one or more reconstructed global latent variables corresponding to one or more components using a flow-based model Each Feedback to the decoder d to generate (435) the reconstructed kth component and the post-reconstruction mask (e.g., the post-reconstruction pixel assignment label) It is m k In one or more embodiments, Based on latent factors In one or more embodiments, the output of the decoder can be used One or more data processing applications (e.g., digital image processing, etc.) used in an inference process, or one or more losses used to train a model in a training process are used to construct (440).

[0135] In one or more embodiments, the message passing prior can suppress the degrees of freedom of the model and can also capture the interactions between different segments or components. According to an embodiment of the present disclosure, Figure 1 The aggregate prior model shown in can be used as Figure 3 The flow-based model in is used for global latent variable encoding and decoding. Figure 3 As shown in 0(1) ,…,z 0(K)] is used as the input of the flow-based model. The global latent variable z0 410 can be inferred by forward message passing (encoding). By backward message delivery (decoding), here, is the reconstructed global latent variable of component k.

[0136] Figure 5 A process for global latent variable encoding and decoding using a flow-based model according to an embodiment of the present disclosure is depicted. For one or more global latent variables z of component k (k=1,...,K) 0(k) Passed through a flow-based model (505) including one or more flow functions to generate one or more transformed global latent variables f k (z 0(k) ), k = 1, ..., K. In one or more embodiments, the one or more stream functions are reversible functions. Each of the one or more stream functions applies a transformation to a global latent variable of a component to generate a corresponding reconstructed global latent variable of the component. The transformed global latent variable f of each component may be k (z 0(k) ) obtains (510) a global latent variable z0 (also referred to as an aggregated global latent variable). In one or more embodiments, the global latent variable z0 is the transformed global latent variable f of all components. k (z 0(k) ), for example, The global latent variable z0 is passed backward through the flow-based model (515) to obtain the reconstructed global latent variable for each component In one or more embodiments, the stream function f k Apply the inverse transform to the global latent variable z0 To obtain the corresponding reconstructed global latent variable of component k

[0137] In one or more embodiments, the global latent variable z0 and the global latent variable z of each component 0(k) The relationship between (k=1,...,K) is used as the encoding (where f={f1,...,f k}) and decode (where ) process. Using flow-based models as encoders and decoders for the second-layer latent variables can limit the degree of freedom of the model and can capture the interactions between different segments or components. By aggregating all reversible functions f k The output of z0 is encoded as Right now by As a decoder The output, It can be expressed as

[0138] 4. ELBO Example for Two-Level Latent Model

[0139] In one or more embodiments, to derive the ELBO, a two-stage variational autoencoder (VAE) with simplified notation is used as a start. Afterwards, the derivation of the model is extended. For the kth component, its first layer of latent variables has two parts z k and z 0(k) . Only z 0(k) is connected to layer 2 z0. Therefore, the ELBO has two components related to these two different potential parts. In one or more embodiments, (x, m k ) as the input of the kth component of the encoder, and (z k ,z 0(k) ) as the first layer latent variable, z0 as the second layer variable, and z 0(k) and As a reformulation, the ELBO of two-level VAE can be given by the following theorem.

[0140] Theorem 2. Let is the ELBO for component k in the two-level split VAW model, then:

[0141]

[0142] The derivation of ELBO is given in the following subsection a) “Derivation of ELBO”. Here, a, e, d, f k are the attention, encoder, decoder and stream functions of component k respectively. In one or more embodiments, the above ELBO (18) with respect to x and m k The reconstruction term of can be given as: In one or more embodiments, the reconstruction loss of a data sample x is obtained by paying attention to the mask (m k ) is weighted. In one or more embodiments, the mask (m k ) follows a Bernoulli distribution parameterized by a Sigmoid function. In one or more embodiments, based on the assumption that the KL divergence between the outputs from the two neural networks (a and d) can be used, the reconstruction loss for the mask (for Φ k The second term in equation (21) is tractable. In one or more embodiments, for x and m k The reconstruction terms of both can be rewritten as:

[0143]

[0144] In one or more embodiments, the regularization term of the first layer latent variables is given by

[0145] Ψ k =-KL(q e (z k |x,m k )‖p(z k ))+H(z 0(k) |x,m k ) (twenty two)

[0146] In one or more embodiments, it is assumed that all latent variables follow a Gaussian distribution. Both the KL and entropy terms can be calculated by reparameterization. Improved dissociation can be achieved using the total correlation (TC) of the component local representations of the KL term in equation (22). In one or more embodiments, the objective function for maximization across all components can be given by the following formula:

[0147]

[0148] a) Export of ELBO

[0149] In one or more embodiments of the present disclosure, a two-stage VAE enhanced by a recursive attention mechanism is disclosed. The ELBO of the model can be optimized. Figure 3 As shown in , the attention network generates the mask m of component k k The input of encoder e is [x k ,m k ], and the corresponding reconstruction generated from decoder d is z′ k =z k z 0(k) is the overall latent variable for component k. See Table 1 for more details on these notations.

[0150] Proof of Theorem 2: In a two-stage autoencoder, (x,m k ) is the input of the first layer, and (z k ,z 0(k) ) is the latent variable of the first layer. At the same time, z 0(k) is also the input of the second layer, and z0 is the latent variable of the second layer. and are the reconstructions of the first and second inputs respectively. The second-level posterior distribution for z0 is q f (z0|z 0(1) z 0(2) ...z 0(K) ).

[0151] The derivation of ELBO starts with a two-level VAE with simplified notation. The derivation extends to embodiments of the model. In one or more embodiments, z l , l∈{1,2} is used to represent the latent variables in layer l. Let z={z 1 ,z 2}, we can get the following equation:

[0152]

[0153]

[0154] In one or more embodiments, the second term in the above equation may be expanded as follows.

[0155]

[0156]

[0157] Therefore, EBLO can be written as:

[0158]

[0159] In one or more embodiments, for the kth component, its first-layer latent variables have two parts z k and z 0(k) . Only z 0(k) is connected to layer 2 (z0). Therefore, the ELBO has two components for these two different latent parts,

[0160]

[0161] Here is e is the posterior distribution of the first layer latent variables parameterized by encoder e. d are x and m parameterized by the decoder d k The distribution of k is a flow-based model of k, and f = {f1,...,f K}. Conditional distribution q f (z0|z 0(k) ) Capture z 0(k) With another z 0(j) (j≠k). In one or more embodiments, it is assumed that all latent variables follow a Gaussian distribution. In one or more embodiments, the posterior q f (z0|z 0(1) z 0(2) …z 0(K) ) is set to a fixed value of 1.

[0162] 5. Implementation of global latent variable reasoning

[0163] In one or more embodiments, for component k, the term of the second layer in the ELBO of equation (20) with respect to the latent variable z0 may be given by:

[0164]

[0165] It can be seen that The calculation of the value involves encoding (q f (z0|z 0(k) )) and decoding (p fk (z 0(k) |z0)) two processes. The first term is to calculate z given z0 0(k) The conditional log-likelihood of all f k The learning of is the same as the learning of the encoder and decoder in the standard VAE model. In one or more embodiments, the prior p(z0) is a standard normal distribution. To calculate the KL divergence, we can use the prior from q f (z0|z 0(k) ) of the samples. In one or more embodiments, the value of the global latent variable can be inferred from any subset of the components, for example, ‖f for each k k (z 0(k) )-z0‖ 2 The value of may be penalized to achieve the stated goal. According to equation (3), the goal It can be finally determined as:

[0166]

[0167] Here, C k is a constant. The KL term can be calculated using the disclosure about equation (2).

[0168] 6. Example of causal direction

[0169] Section 5 above discloses using the learned aggregate prior model to infer the relationship between the two components. By maximizing the objective (ELBO) of the autoencoder in the second layer, we get the objective In one or more embodiments, maximizing this objective may force each component to have the same potential value, i.e., z0=f1(z 0(1) )=...=f K (z 0(K) ). In one or more embodiments, z 0(i) and z 0(j) It can be used to infer the causal interaction between components i and j. Use z 0(i) Prediction z 0(j) The value of Use z 0(j) Prediction z 0(i) The causal direction can be determined using the variance of the prediction errors by following the regression error-based causal inference (RECI) approach.

[0170] In one or more embodiments, assuming that the two data components are a pair of cause variables and effect variables, the causal relationship can be detected by extending the cause-effect reasoning method. α =φ(c)+αn, where α is a positive real number and n follows a noise distribution, then based on Theorem 3, the expected variance of the prediction error is used to reveal the causal direction. In one or more embodiments, according to the theory of Theorem 3, the ratio of the expected variance of the prediction error is used as the score that reveals the causal direction of a pair of variables.

[0171] Theorem 3. For a pair of cause-effect variables {c, e}, e α =φ(c)+αn, under assumptions (a)-(d), the limit Established.

[0172] (a) φ:[0,1]→[0,1] is reversible, monotonic, and twice differentiable;

[0173] (b) Tight support;

[0174] (c) as well as as well as

[0175] (d)Cov[φ′,Var[n|c] pc ]=0.

[0176] pass Figure 2 The well-trained flow-based model shown in (left) can be used to convert z to 0(k) The value of is estimated for any z 0(j) , j≠k. In one or more embodiments, the following theorem on causal detection may be utilized. Theorem 3 provides a method for detecting causal relationships between components. The proof of Theorem 3 and more details on component interaction detection are described below.

[0177] Proof of Theorem 3: For two random variables and Given by g The conditional variance of definition. When Reach g when reaching A random variable with a probability of . Its expectation is given by:

[0178]

[0179] For a reversible function h, we can obtain the equation Var[h(G)|g]=0 and

[0180] In one or more embodiments, it can be observed that

[0181]

[0182] Therefore, there may be

[0183]

[0184] In the subsequent steps of the above equation, due to By α is uniformly bounded, αn + and αn- vanish in the limit, and the variance is bounded by 1. is uniformly bounded. Therefore, the bounded convergence theorem states:

[0185]

[0186] Using Taylor's theorem, we can get:

[0187]

[0188] here is a value in [e-αn,e]. In addition, we can have

[0189] Therefore, there may be

[0190]

[0191] The last term in equation (34) can be rewritten as:

[0192]

[0193] Here, the above inequality is based on the Cauchy-Schwarz inequality. If φ is linear, the last term of the above formula becomes 1, because φ′=1. Alternatively, equation (35) can be stated to complete the proof of Theorem 3.

[0194]

[0195]

[0196] Figure 6The process of inferring a causal relationship between two components using a trained flow-based model according to an embodiment of the present disclosure is depicted. In one or more embodiments, maximizing the objective in equation (28) forces (605) each component to have the same potential value, i.e., z0=f1(z 0(1) )=…=f K (z 0(K) ).

[0197] In one or more embodiments, z 0(i) and z 0(j) Reasoning about a causal interaction between a first component (e.g., component i) and a second component (e.g., component j) of an input (e.g., an image). Using z 0(i) Predict (610) the reconstructed global latent variable value of component j Similarly, you can Using z 0(j) Predict (615) the reconstructed global latent variable value of component i In one or more embodiments, the causal direction between component i and component j may be determined (620) using the expected variance of the prediction errors of the two predictions. According to Theorem 3, if (Use z 0(i) The expected variance of the forecast error of Greater than (Use z 0(j) The expected variance of the forecast error of Component i is identified as the cause and component j is identified as the effect.

[0198] 7. Example of dissociation with TC

[0199] In one or more embodiments, for each component, the local latent variable z in equation (3) in Section C.2 is k The KL term of can be rewritten as:

[0200]

[0201] Here, z ki Yes k The i-th term of . In one or more embodiments, the total correlation (TC) is penalized to force the dissociation of local latent factors. In one or more embodiments, to calculate the second term, a weighted method can be used to estimate the distribution value q(z).

[0202] D. Experimental results

[0203] Embodiments of the present disclosure were evaluated using synthetic and real-world data. For the synthetic data, simulations were performed using a multi-object setting. Using this dataset, it was demonstrated that embodiments of the present disclosure can outperform other methods when correlations exist between objects. Those embodiments were further validated using some real-world data. In one or more experiments, the causal relationship dataset was also used to evaluate the component interaction discovery of the model.

[0204] It should be noted that these experiments and results are provided as illustrations only and were conducted under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of the current patent document.

[0205] 1. Performance indicators

[0206] Some of the experiments focus on dissociation, segmentation and component interactions as well as performance comparison with MONet.

[0207] Dissociation. Dissociation evaluation metrics have been proposed previously. Someone defined a metric that exploits the accuracy that a low VC dimension linear classifier can achieve when identifying a fixed ground truth factor. The disadvantage of this method is the lack of axis alignment detection. It has been proposed to use the mutual information gap (MIG) between latent variables and ground truth factors to measure dissociation. In one or more embodiments of the present disclosure, regression-based methods are used in various experiments. The regression-based method divides the latent space data into training, evaluation, and testing. A dissociation score is obtained based on the performance of the learned regression model.

[0208] Segmentation In one or more experiments, segmentation is evaluated using the adjusted rand index (ARI). The ground truth masks and predicted masks are converted to binary values, and the similarity of a pair of masks is based on the number of identical item values. In one or more experiments, the ARI score can be calculated using a pairwise similarity matrix.

[0209] Component interaction causality can refer to a relationship between two events, where one event (result) precedes the other event (cause). In one or more experiments, the methods disclosed in Section C.5 are used for correlation and causal component interaction discovery. The correlation baseline in the experiment uses MONet for latent representation learning and is tested independently using the Hilbert-Schmidt Independence Criterion (HSIC). In order to make a fair comparison, the disclosed model (also referred to as "CONet" below) and the embodiments of the baseline model (MONet) use the same network structure for the encoder, decoder, and attention network. Detailed information on network structure embodiments can be found in Section E. Comparisons with existing methods are also performed on benchmark data. In the supplementary file, more results on other datasets are provided.

[0210] 2. Simulated Multi-Object Dataset

[0211] In this section, embodiments of the present disclosure are evaluated using a simulated 2-object dataset that includes images generated with three types of objects (green squares, red circles, and blue diamonds). Multiple samples are used for training, and multiple samples are used for testing. Figure 7 Some exemplary sample images are shown in the first row of . For the results in this subsection, a LASSO regressor with α=0.2 was used for the dissociation score. Object relations were incorporated into the dataset to evaluate the performance of different models.

[0212] In the first set of experiments, images containing two objects were generated. Only object pairs {circle, circle}, {circle, square}, {square, square}, and {square, diamond} appeared in the same image. Circle and diamond did not appear in the same image. γ was set to 0.5 for both models, while β values ​​for both MONet and our model were adjusted according to the dissociation and segmentation scores. Figure 7 Figure 2 shows the original image, reconstructed image, and mask image from MONet and the model embodiment of the present disclosure (“proposed”) after 50 iterations. Due to the interference of object relations, MONet cannot distinguish between diamonds and circles, such as Figure 7As shown in . In addition, the model implementation is studied with more complex object relationships. In this set of generated images, each image has two or three objects. Similarly, a circle and a square, a square and a rhombus can also appear in one image. Circles and rhombuses are not allowed to appear in the same image. The experimental results show that this method can effectively dissociate the structured latent factors. It is also noted that the structured latent space with TC penalty can also improve the dissociation score of MONet. More results of this set of experiments are shown below.

[0213] More results on the simulated dataset:

[0214] Figure 8 The dissociation scores (dscore) for different models and different numbers of epochs are given. Figure 8 In the figure, there are four bars for each epoch, corresponding to "Proposed", MONet, MONet-S, and MoNet-ST (starting from left to right), and all epochs have the same order. "MONet-S" refers to MONet with structured latent variables introduced in the dual-level latent structure section of the main paper. "MONet-ST" refers to MONet with structured latent variables in addition to total correlation (TC). The left figure shows the dissociation scores (D-Scores) of different methods on the simulated 2-object dataset, and the right figure gives the D-Scores on the 3-object dataset. In the 3-object dataset, each image contains two or three objects. Similar to the 2-object dataset, a circle and a square, a square and a diamond may appear in one image, while a circle and a diamond are not allowed to appear in the same image. These rules are the latent component relationships of the dataset.

[0215] like Figure 8 As shown in the 2-object dataset figure in , the proposed aggregation prior via message passing can effectively capture the latent factor structure and improve the dissociation score on the 2-object dataset. Figure 8 The 3-object dataset figure in depicts the dissociation scores of different models on a simulated 3-object dataset. It can be seen that embodiments of the disclosed method can effectively dissociate the structured latent factors. It is also noted that the structured latent space with TC penalty can improve the dissociation score of MONet.

[0216] 3. Evaluation Dataset

[0217] In one or more experiments, embodiments of the present disclosure were evaluated using an evaluation dataset. Each image in the evaluation dataset may include one or more shapes with a background. In one or more experiments, all available features for dissociation testing were used for evaluation. These features may include position (x and y), shape, color (e.g., RGB values), orientation, scale, visibility (a binary feature indicating whether the object is not empty). The dissociation score can be calculated using LASSO as a regressor and α=0.2. In one or more experiments, γ=0.5 for both models and β for both methods were adjusted. The dissociation and segmentation performance after 20 epochs is given in Table 2, with a learning rate of 10 for both models. -4 . Table 2 shows that the embodiments of the present disclosure can achieve excellent dissociation scores and segmentation scores. More results of the segmentation scores (ARI) of the two methods are presented in the supplementary file. It can be seen that the embodiments of the present disclosure can consistently improve the segmentation together with the update step. The embodiments of the present disclosure can produce more reasonable object segmentation visually.

[0218] Table 2: Dissociation scores and segmentation scores of MONet and the proposed method (“Proposed”) tested on the evaluation dataset.

[0219]

[0220] Other results on the evaluation dataset:

[0221] Fig. 9 The values ​​of segmentation scores (ARI) for embodiments of the present disclosure and MONet over epochs are given. For each epoch, there are two bars corresponding to "Proposed" and "MONet" (from left to right). The order is the same for all epochs. It can be seen that, using the adjustment information between components, embodiments of the present disclosure can continuously improve the segmentation with more epochs.

[0222] 4. Polyomino Dataset

[0223] In one or more experiments, multiple images from the Polyomino dataset were used. Each Polyomino image can include several Polyominos, such as tetromino, sampled from multiple different shapes or directions. There are four components for MONet and the embodiments of the present disclosure. Multiple images are randomly selected to evaluate the dissociation and segmentation scores, and the two models are trained using multiple images. The experimental process follows the previous two datasets. Table 3 gives the dissociation scores and segmentation scores of the two methods after 1 period (α of the dissociation score = 0.1). It can be seen that the embodiments of the present disclosure can improve both segmentation and dissociation. Fig.10The segmentation ARI scores of the MONet model 1005 and the model embodiment 1010 (“proposed”) over iterations are given, showing that the model 1010 can achieve successively higher segmentation scores. A large number of combinations of different colors and shapes result in lower segmentation scores. Clearly, the proposed method can segment objects very well.

[0224] Table 3: Dissociation scores and segmentation scores of MONet and method embodiment (“proposed”) tested on Polyomino.

[0225]

[0226] 5. Results of component interaction detection

[0227] In this set of experiments, a cause-effect benchmark dataset was prepared using the Causal Discovery Toolbox. Unlike existing causal discovery methods, in one or more embodiments, each variable in a cause-effect pair is tiled into a component in an image, and the causal relationship is learned using a latent representation. Since each component contains only one variable, the variable segmentation ability of the model can be tested using a synthetic image. In one or more experiments, simple cause-effect pairs are considered, and the causal direction is ignored. Several pairs of variables with a relationship are tiled into component pixels to form an image. The synthesized image is then passed into our network. Similar to previous experiments, the output includes reconstructed components and component masks. Similarly, an ARI-based m-score is reported to show whether each reconstructed component has been correctly discovered and segmented. In addition, one or more experiments are conducted to evaluate whether component pairs with causal relationships can also be discovered.

[0228] In one or more experimental settings, a kernel-based nonparametric independence test is used to score the relationship between a pair of components for MONet with the Hinet-Schmidt independence criterion (HSIC). In one or more embodiments of the disclosed model (CONet), the conditional probabilities of the latent variables in Section C.4 are used for relationship learning. As shown in Table 4, an embodiment of CONet ("proposed") outperforms MONet by 1.0% in terms of dissociation score. It is worth noting that an embodiment of CONet can find a larger number of correct relationship pairs than MONet. Such results indicate that in addition to component segmentation, embodiments of CONet have stronger capabilities in causal relationship discovery. Section D.7 provides more results on component interaction detection.

[0229] Table 4: Dissociation scores and number of causal pairs found.

[0230]

[0231] 6. Facial image test results

[0232] A number of facial images are randomly selected for training and testing. These facial images include multiple attributes, including gender, hair color, whether wearing glasses, etc. In one or more experiments, at least some of the attributes are used to evaluate the dissociation of MONet and embodiments of the present disclosure. Table 5 gives the dissociation scores of the two models (α=0.3). Fig.11 The graph in gives the loss (loss, -logP, KL mask , and -L f ) at the iteration. Fig.11 The upper left graph in is the negative ELBO value.

[0233] Table 5: Dissociation scores of MONet and method embodiment ("Proposed").

[0234]

[0235] 7. Results of the causal direction test

[0236] The setup for this set of experiments may be similar to Section D.5. In one or more experiments, the cause-effect benchmark dataset “tuebingen” prepared using the Causal Discovery Toolbox is used. Four pairs of causal effect variables are taken and each variable is tiled into components of the image in the cause-effect pair. The latent representation is used to learn the causal relationships. Since each component contains only one variable, the latent z of the component is tiled. 0(k) Used for causal detection.

[0237] The embodiments of the present disclosure are compared with the MONet+RECI method. For the MONet+RECI method, MONet is used to learn the latent representation of each component, and then the variance of the regression error between the latent representations is used to determine the causal direction. In this baseline method, the implementation of RECI in the Causal Discovery Toolbox is used. For the embodiments using CONet, the scores defined in Theorem 3 are calculated by the aggregate model based on the prediction errors, and then these scores are used to determine the causal direction.

[0238] To compute the accuracy of causal detection, the ground truth labels of each component are obtained from the model by comparing its mask with the ground truth mask during the data synthesis stage. After thresholding using a value γ, a list of component pairs with causal relationship scores is obtained. Then, the percentage of correct causal direction pairs is calculated. Table 6 gives the accuracy of correct causal pairs with different thresholds. It can be seen that the disclosed framework embodiments always learn better representations for causal detection.

[0239] Table 6: Causal detection accuracy (%) on 'tuebingen' at different thresholds.

[0240]

[0241] E. Network Structure Example

[0242] This section discloses some network structure embodiments for encoder and decoder, which are shown in Table 7 and Table 8 respectively. Note that the network adopts a U network with 5 blocks. The decoder is a spatial broadcast decoder to encourage VAE to learn spatial features.

[0243] Table 7: Network structure example of encoder. dim Yes k The length of plus the length of z0.

[0244]

[0245]

[0246] Table 8: Network structure example of decoder.

[0247]

[0248] F. Some conclusions

[0249] In the present disclosure, embodiments of a novel two-level framework are disclosed to learn structured latent factors for dissociation. In one or more embodiments, flow-based structural priors of the latent representation enable the model to learn interactions between components through a message passing scheme. The framework improves existing scene segmentation methods on both dissociation and segmentation. It is shown that the framework embodiments can capture internal interactions between data components in experiments.

[0250] Those skilled in the art will appreciate that the present disclosure can be applied to various scenarios, for example, physical interaction extraction. Physical interactions between objects are important common sense or prior knowledge for humans to make feasible decisions. Objects placed in a static scene usually adhere to certain relationships, such as pens and paper, books and bookcases, cups and tables, etc. Another useful application is to combine the embodiments of the present disclosure with reinforcement learning. By using the learned relationships between objects, the search space of the agent can be greatly reduced and reasonable decisions can be made more efficiently.

[0251] In one or more embodiments of the present disclosure, data segmentation and representation learning are integrated by developing a two-level VAE framework. Using reasoning methods, the two-level VAE framework can learn more meaningful data structure representations. In addition to the data sets presented in the experiment, the framework can also be applied to other types of data. The embodiments of the present disclosure can potentially expand the application of unsupervised learning and self-supervised learning to a wider range of scenarios, such as information extraction, knowledge discovery, etc.

[0252] G. Computing System Embodiments

[0253] In one or more embodiments, aspects of the present patent document may be directed to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any tool or aggregation of tools operable to compute, calculate, determine, classify, process, send, receive, retrieve, initiate, route, switch, store, display, communicate, display, detect, record, copy, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., a laptop), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smart phone, a tablet phone, a tablet computer, etc.), a smart watch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. The computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more disk drives, one or more network ports for communicating with external devices and various input and output (I / O) devices, such as a keyboard, mouse, stylus, touch screen, and / or video display. A computing system may also include one or more buses operable to transmit communications between various hardware components.

[0254] Fig.12 A simplified block diagram of an information processing system (or computing system) according to an embodiment of the present disclosure is depicted. It will be appreciated that the functionality shown for system 1200 may operate to support various embodiments of the computing system - although it should be appreciated that the computing system may be configured differently and include different components, including having fewer or more components, such as Fig.12 as shown in .

[0255] like Fig.12 As shown in , computing system 1200 includes one or more central processing units (CPUs) 1201, which provide computing resources and control the computer. CPU 1201 can be implemented with a microprocessor or the like, and can also include one or more graphics processing units (GPUs) 1202 and / or floating point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 1202 can be incorporated into a display controller 1209, such as part of one or more graphics cards. System 1200 can also include system memory 1219, which can include RAM, ROM, or both.

[0256] like Fig.12 As shown in , multiple controllers and peripherals may also be provided. Input controller 1203 represents an interface to various input devices 1204 such as a keyboard, a mouse, a touch screen, and / or a stylus. The computing system 1200 may also include a storage controller 1207 for interfacing with one or more storage devices 1208, each storage device 1208 including a storage medium such as a tape or a disk, or an optical medium that can be used to record a program of instructions for an operating system, a utility program, and an application, which may include an embodiment of a program that implements various aspects of the present disclosure. According to the present disclosure, the storage device 1208 may also be used to store processed data or data to be processed. The system 1200 may also include a display controller 1209 for providing an interface to a display device 1211, which may be a cathode ray tube (CRT) display, a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 1200 may also include one or more peripheral device controllers or interfaces 1205 for one or more peripheral devices 1206. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. Communications controller 1214 may interface with one or more communications devices 1215, which enables system 1200 to connect to remote devices via any of a variety of networks including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN), or via any suitable electromagnetic carrier signals including infrared signals. As shown in the depicted embodiment, computing system 1200 includes one or more fans or fan trays 1218 and one or more cooling subsystem controllers 1217 that monitor the thermal temperature of system 1200 (or its components) and operate fans / fan trays 1218 to help regulate the temperature.

[0257] In the illustrated system, all major system components may be connected to bus 1216, which may represent more than one physical bus. However, various system components may or may not be physically close to each other. For example, input data and / or output data may be remotely transmitted from one physical location to another physical location. In addition, programs that implement various aspects of the present disclosure may be accessed from a remote location (e.g., a server) over a network. Such data and / or programs may be transmitted via any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and tapes; and optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0258] Aspects of the present disclosure may utilize instructions for one or more processors or processing units to cause steps to be performed to be encoded on one or more non-transitory computer-readable media. It should be noted that one or more non-transitory computer-readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. ASICs, programmable arrays, digital signal processing circuits, etc. can be used to implement the functions of hardware implementations. Therefore, the term "device" in any claim is intended to cover software and hardware implementations. Similarly, the term "computer-readable medium" used herein includes software and / or hardware, or a combination thereof, on which a program including instructions is included. In view of these alternatives to implementation, it should be understood that the drawings and the accompanying descriptions provide the functional information required for those skilled in the art to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) to perform the desired processing.

[0259] It should be noted that embodiments of the present disclosure may further relate to computer products having non-transitory tangible computer-readable media having computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be those known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specially configured to store or store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code, such as machine code generated by a compiler and files containing higher-level code executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine executable instructions, which may be in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located locally, remotely, or both.

[0260] Those skilled in the art will recognize that no computing system or programming language is essential for the practice of the present disclosure. Those skilled in the art will also recognize that many of the above elements can be physically and / or functionally separated into modules and / or sub-modules or combined together.

[0261] Those skilled in the art will appreciate that the foregoing examples and embodiments are exemplary and do not limit the scope of the present disclosure. It is intended that all permutations, enhancements, equivalents, combinations and improvements that are apparent to those skilled in the art through reading the specification and studying the drawings are included within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged differently, including having multiple dependencies, configurations and combinations.

Claims

1. A method for learning latent structural relationships, comprising: receiving an input comprising one or more components, wherein the input is an image, each of the one or more components in the image represents an object in the image and an element of the input represents a pixel in the image; Using the attention network, generating a mask corresponding to each of the one or more components, the mask indicating a probability that at least a subset of elements of the input belongs to each of the one or more components; Using an encoder, the input and each mask are encoded into latent variables, where the latent variables include global latent variables and local latent variables of the corresponding components; transforming the one or more global latent variables into one or more transformed global latent variables using a flow-based model including one or more flow functions, each flow function forward transforming one of the one or more global latent variables to obtain a corresponding transformed global latent variable; generating an aggregated global latent variable based on the one or more transformed global latent variables; transforming the aggregated global latent variable into one or more reconstructed global latent variables corresponding to the one or more components using a flow-based model; generating, using a decoder, a pair of a reconstructed mask and a reconstructed component for each of the one or more components by using a reconstructed global latent variable and a local latent variable corresponding to each of the one or more components; and The reconstructed components and the reconstructed masks corresponding to the one or more components are used for one or more data processing applications in an inference process, or for constructing one or more losses for training in a training process.

2. The method according to claim 1, wherein: The aggregated global latent variable is the average of one or more transformed global latent variables corresponding to one or more components.

3. The method according to claim 1, wherein: One or more stream functions are reversible functions.

4. The method of claim 3, wherein: Each reconstructed global latent variable is obtained by backward transforming the aggregated global latent variable using the inverse function of the corresponding stream function.

5. The method of claim 4, wherein: One or more stream functions of the flow-based model are forced to generate one or more global latent variables that have the same value for one or more components by maximizing an evidence lower bound ELBO of the flow-based model.

6. The method according to claim 5, further comprising Obtaining a prediction of a reconstructed global latent variable value for a second component of the one or more components via a backward transformation of the transformed global latent variable for a first component of the one or more components; Obtaining a prediction of a value of the reconstructed global latent variable of the first component by backward transforming the transformed global latent variable of the second component; and A causal direction between the first component and the second component is determined using an expected variance of a prediction error for predicting a reconstructed global latent variable value of the second component and an expected variance of a prediction error for predicting a reconstructed global latent variable value of the first component.

7. The method according to claim 6, wherein: In response to an expected variance of prediction error for predicting the reconstructed global latent variable value of the second component being greater than an expected variance of prediction error for predicting the reconstructed global latent variable value of the first component, it is determined that the first component is a cause and the second component is an effect.

8. The method according to claim 1, wherein: A mask corresponding to each of the one or more components is sequentially generated based on the input and a range of each component, the range of each component being derived based on a range corresponding to a previously processed component and the mask.

9. A system for learning latent structural relationships, comprising: one or more processors; as well as A non-transitory computer-readable medium comprising one or more sets of instructions, which, when executed by at least one of one or more processors, cause the at least one processor to perform the method according to any one of claims 1-8.

10. A non-transitory computer readable medium comprising one or more instruction sequences which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1-8.

11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images