Apparatus and system for unsupervised entanglement disentangling representation learning with residual variational autoencoder

Abstract

Apparatuses and systems are provided for unsupervised entanglement disentangling representation learning with residual variational autoencoders. A computer program product storing instructions that, when executed by a computer, cause the computer to receive input data, encode the input via an encoder, obtain, during a first sequence, a first latent variable defining a property of the input data, generate a sequential reconstruction of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstruction with a comparison of at least the first latent variable, and output a final reconstruction of the input data with a plurality of residuals from a plurality of sequences.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to augmentation and image processing of images with autoencoders. BACKGROUND

[0002] Deep convolutional neural networks (CNNs) can have a tendency to learn surface statistics of data associated with a given task rather than important generative factors underlying in the physical world. For example, a system can be trained to identify an animal (e.g., a cat) rather than important factors (e.g., whiskers, unique ears, unique eyes, etc.) that can be associated with the cat. SUMMARY

[0003] According to one embodiment, a system comprises a camera configured to obtain image information from an object, and a processor in communication with the camera and programmed to receive input data comprising the image information, encode the input via an encoder, obtain, during a first sequence, a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, and output a final reconstruction of the input data with a decoder and a plurality of residuals from a plurality of sequences.

[0004] According to another embodiment, an apparatus comprises a processor programmed to receive input data, encode the input via an encoder, obtain, during a first sequence, a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, and output a final reconstruction of the input data with a decoder and a plurality of residuals from a plurality of sequences.

[0005] According to a last embodiment, a computer program product storing instructions that, when executed by a computer, cause the computer to receive input data, encode the input via an encoder, obtain, during a first sequence, a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, and output a final reconstruction of the input data with a decoder and a plurality of residuals from a plurality of sequences. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 A block diagram of a computer system is disclosed.

[0007] Figure 2 Flowcharts of a residual variational autoencoder (VAE) and a residual learning software application are disclosed.

[0008] Figure 3 A block diagram of a residual variational autoencoder and residual learning software application is disclosed.

[0009] Figure 4 Examples of latent traversals across each latent dimension for an example dataset are disclosed. DETAILED DESCRIPTION

[0010] Recent advances in unsupervised disentangled representation learning have focused on extending variational autoencoders (VAEs) with a disentangling objective function to balance the trade-off between disentanglement and reconstruction due to the information bottleneck of VAEs. A residual variational autoencoder (Res-VAE) can include a "residual learning" mechanism as a training regime instead of a disentangling objective function. It can learn from the residual between the input data and the cumulative reconstruction of sequentially added latent variables. A disentangled representation can be one in which changes in one dimension correspond to changes in only one generating factor of the data, while being relatively invariant to changes in other factors. This formulation can encourage disentanglement in the latent space by introducing explicit correlation structures and relax the network reconstruction by decomposing it into the decoder output and a residual. Res-VA can eliminate the hyperparameter tuning of the disentangling objective, which is a critical step for state-of-the-art performance.

[0011] Learning sparse and interpretable representations of data is a crucial component of generalizable, robust, and explainable intelligent systems. Despite recent advances in representation learning, deep convolutional neural networks (CNNs) have been shown to have a tendency to learn superficial statistics of the data associated with a given task rather than important generating factors embedded in the physical world. One way to move towards this is disentangled representation learning, which aims to capture independent and interpretable generating factors of the data. A disentangled representation can be one in which changes in one dimension correspond to changes in only one generating factor of the data, while being relatively invariant to changes in other factors.

[0012] Unentangled representations can be learned in a supervised manner where explicit prior knowledge about the generating factors of the data and / or implicit prior knowledge is available. However, it can be desirable to learn this in an unsupervised learning setting to leverage the large amount of available unlabeled data. With the recent development of generative models, a number of unentangled learning schemes have been proposed based on generative adversarial networks (GANs) or variational autoencoders (VAEs). Unsupervised learning of unentangled representations is not possible without introducing inductive bias on the model or the data. In VAE-based schemes, the trade-off between unentanglement and reconstruction can be addressed by different training schemes. The idea of using different training schemes instead of augmenting the objective function is similar to using different methods to prevent overfitting of the model. For example, one common way to address the problem of overfitting is to augment the objective function with a regularization term such as L 1 or L 2 regularization). Another way is to apply special operations during training to enforce generalization of the network representation, such as Dropout or batch normalization.

[0013] Res-VAEs can leverage a“residual learning” mechanism to learn unentangled representations. The“residual learning” mechanism can include explicit correlation structure between latent variables that encourages unentanglement. Res-VAE systems can decompose the reconstruction into a residual and the network output. This relaxation of the decoder reconstruction can reduce the trade-off between unentanglement and reconstruction and improve unentangled representation learning.

[0014] Figure 1 A data annotation system 100 implementing a system for annotating data is depicted. The data annotation system 100 can include at least one computing system 102. The computing system 102 can include at least one processor 104 operatively connected to a memory unit 108. The processor 104 can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 106. The CPU 106 can be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, the CPU 106 can execute stored program instructions retrieved from the memory unit 108. The stored program instructions can include software that controls the operation of the CPU 106 to implement the operations described herein. In some examples, the processor 104 can be a system on a chip (SoC) that integrates the functionality of the CPU 106, the memory unit 108, network interfaces, and input / output interfaces into a single integrated device. The computing system 102 can implement an operating system for managing various aspects of the operation.

[0015] The memory unit 108 can include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory can include solid state memory such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 102 is deactivated or loses power. The volatile memory can include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 108 can store a machine learning model 110 or algorithm, a training data set 112 for the machine learning model 110, an original source data set 115.

[0016] The computing system 102 can include a network interface device 122 configured to provide communication with external systems and devices. For example, the network interface device 122 can include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 122 can include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 122 can be further configured to provide a communication interface to an external network 124 or cloud.

[0017] The external network 124 can be referred to as the World Wide Web or the Internet. The external network 124 can establish standard communication protocols between computing devices. The external network 124 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 130 can be in communication with the external network 124.

[0018] The computing system 102 can include an input / output (I / O) interface 120 that can be configured to provide digital and / or analog input and output. The I / O interface 120 can include additional serial interfaces (e.g., Universal Serial Bus (USB) interfaces) for communicating with external devices.

[0019] The computing system 102 can include a human-machine interface (HMI) device 118 that can include any device that enables the system 100 to receive control inputs. Examples of input devices can include human interface inputs such as a keyboard, mouse, touch screen, voice input device, and other similar devices. The computing system 102 can include a display device 132. The computing system 102 can include hardware and software for outputting graphical and textual information to the display device 132. The display device 132 can include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user or operator. The computing system 102 can be further configured to allow interaction with remote HMIs and remote display devices via the network interface device 122.

[0020] The system 100 can be implemented using one or more computing systems. While the example depicts a single computing system 102 that implements all of the described features, it is intended that various features and functionality can be separated and implemented by multiple computing units in communication with each other. The particular system architecture chosen can depend on a variety of factors.

[0021] The system 100 can implement a machine learning algorithm 110 configured to analyze a raw source dataset 115. The raw source dataset 115 can include raw or unprocessed sensor data that can represent an input dataset for a machine learning system. The raw source dataset 115 can include video, video clips, images, text-based information, and raw or partially processed sensor data (e.g., radar plots of objects). In some examples, the machine learning algorithm 110 can be a neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm can be configured in a car application to identify pedestrians in video images.

[0022] The computer system 100 can store a training dataset 112 for the machine learning algorithm 110. The training dataset 112 can represent a collection of previously constructed data used to train the machine learning algorithm 110. The machine learning algorithm 110 can use the training dataset 112 to learn weighting factors associated with the neural network algorithm. The training dataset 112 can include a collection of source data with corresponding outcomes or results that the machine learning algorithm 110 attempts to replicate via a learning process. In this example, the training dataset 112 can include source videos with and without pedestrians as well as corresponding presence and location information. The source videos can include a variety of scenarios in which pedestrians are identified.

[0023] The machine learning algorithm 110 can operate in a learning mode using the training dataset 112 as input. The machine learning algorithm 110 can be executed over multiple iterations using data from the training dataset 112. With each iteration, the machine learning algorithm 110 can update internal weighting factors based on the results achieved. For example, the machine learning algorithm 110 can compare the output results (e.g., annotations) with those included in the training dataset 112. Since the training dataset 112 includes expected results, the machine learning algorithm 110 can determine when performance is acceptable. After the machine learning algorithm 110 achieves a predetermined level of performance (e.g., 100% consistent with the outcomes associated with the training dataset 112), the machine learning algorithm 110 can be executed using data that is not in the training dataset 112. The trained machine learning algorithm 110 can be applied to new datasets to generate annotated data.

[0024] The machine learning algorithm 110 can be configured to identify a particular feature in the raw source data 115. The raw source data 115 can include a plurality of instances or input datasets for which an annotation of a result is desired. For example, the machine learning algorithm 110 can be configured to identify the presence of a pedestrian in a video image and annotate the occurrence. The machine learning algorithm 110 can be programmed to process the raw source data 115 to identify the presence of a particular feature. The machine learning algorithm 110 can be configured to identify a feature in the raw source data 115 as a predetermined feature (e.g., a pedestrian). The raw source data 115 can be derived from a variety of sources. For example, the raw source data 115 can be actual input data collected by the machine learning system. The raw source data 115 can be machine generated for testing the system. As an example, the raw source data 115 can include raw video images from a camera.

[0025] In an example, the machine learning algorithm 110 can process the raw source data 115 and output an indication of the image representation. The output can also include an augmented representation of the image. The machine learning algorithm 110 can generate a confidence level or factor for each output generated. For example, a confidence value that exceeds a predetermined high confidence threshold can indicate that the machine learning algorithm 110 is confident that the identified feature corresponds to the particular feature. A confidence value that is less than a low confidence threshold can indicate that the machine learning algorithm 110 has some uncertainty as to the presence of the particular feature.

[0026] A VAE can be a deep directed graphical model composed of an encoder and a decoder. The encoder can map data x to a latent representation and the decoder maps the latent representation back to the data space where and represent model parameters. The loss function of a VAE can be defined as follows:

[0027]

[0028] where can be the Kullback-Leibler divergence between the true and the approximate posterior. By utilizing a prior over the latent representation where is the identity matrix, the VAE can learn a latent representation In one example, the images can be utilized, and the latent representation can be attributes that explain facial features. For example, one coordinate can include the gender of the face (e.g., male or female), whether the face has a certain expression (e.g., happy or sad), whether the face is old or young (e.g., whether the face has wrinkles), etc. disentangled representation learning can be used to identify the latent representation where each latent variable only corresponds to one of the generating factors of the given data x To achieve this, β the VAE utilizes adjustable hyperparameters β to scale the VAE objective, such as:

[0029] (2)

[0030] When , β the VAE can become equivalent to the original VAE framework. β The addition of can encourage the posterior to match the factorized unit Gaussian prior z This can enhance the independence between the latent variables of z , thereby disentangling the representation and decomposing the features that can be generated from the images. On the other hand, it can reduce the amount of information stored in x about β , which can lead to poor reconstructions, especially for high values.

[0031] Table 1 is an example of a residual learning algorithm, as follows:

[0032] .

[0033] In one embodiment, Res-VAE can define an explicit correlation structure between the latent variables. Such a structure can be viewed as an inductive bias on the model. As explained further below, in cases where the focus is on disentanglement, the latent space structure can reduce the solution space. Furthermore, Res-VAE can decompose the reconstruction into the network output and a residual. This can reduce the constraints on the network reconstruction, thereby reducing the trade-off between reconstruction and disentanglement. Res-VAE can eliminate the hyperparameter tuning of the above-mentioned schemes by training with the original VAE objective.

[0034] Res-VAE can include an encoder network labeled and a decoder network labeled Thus, x and may represent the input data and the latent representation vector, respectively. and Network parameters can be represented. Instead of a standard forward pass that updates all latent variables simultaneously, Res-VAE can use a "residual learning" mechanism to update latent variables sequentially based on the residual between the input and the cumulative reconstruction.

[0035] Given the dimensionality of the latent space and the latent representation The "residual learning" mechanism can include d forward passes. Each forward pass can update one latent variable at a time. In the first pass, the input data x is passed through the encoder to compute the parameterized posterior and , where the superscript denotes the pass number. Instead of drawing samples for all latent variables at once, we only sample the first latent variable and set the remaining latent variables to 0. The modified latent variables are then passed through the decoder to generate an output, denoted . We take the difference between the input x and this output , denoted the residual , as the input for the second pass. For the second pass, is passed through the encoder to generate a new parameterized posterior and . This time, we utilize this parameterized posterior to sample only the second latent variable, denoted . We re-sample the first latent variable using while setting the remaining latent variables to 0. The modified latent variables are then used to generate a new reconstruction . We then compute the corresponding residual as the input for the third pass. In the i pass, the i latent variable is sampled from the encoding of the current residual , so . The previously updated latent variables follow their corresponding residual encodings, and the subsequent latent variables are set to zero. The latent variables are passed through the decoder and generate a new residual . This process repeats d times, such that all latent variables are sampled. In one example, when the latent dimensionality is 2, Res-VAE can not produce optimal results. In this case, the residual can compensate for any decoder output and produce a zero reconstruction error.

[0036] Res-VAE can be able to train with the original VAE objective defined in Equation 1. The parameters are updated using standard backpropagation. The prior can be set to be the isotropic unit Gaussian and the posterior can be parameterized as a Gaussian with a diagonal covariance matrix. The “reparameterization” trick can be used to transform each random variable into a differentiable transformation of a noise variable where .

[0037] The residual learning mechanism can define the sequential update of the latent variables. It can define the correlation between the latent variables in the posterior, which can be written as The KL loss term of the original VAE objective in Equation 1 encourages the posterior to match the factorized unit Gaussian prior . By adding explicit correlation through the “residual learning” mechanism, the Res-VAE objective can be seen as a modified VAE objective:

[0038] minimizing

[0039] subject to

[0040] These constraints encourage the newly added latent variables to be independent of the already added latent variables, thus enhancing the disentangled latent representation. Moreover, the solution space of Equation 3 is a subset of the original VAE. By reducing the search space, we can reach the solution space faster and be less sensitive to initialization. In the experimental section, we show that Res-VAE is less sensitive to initialization compared to other methods.

[0041] Res-VAE can sample from the corresponding residual encoding of the latent variables for each latent variable. However, other VAE-based approaches can generate a reconstruction by sampling from the joint distribution of all latent variables. Therefore, Res-VAE can generate a sequence of images during the forward pass. Such images can reflect the image transformation corresponding to the disentangling factor. Res-VAE can update one latent variable at a time in a sequential order (e.g., sequential forward pass). Each latent variable can aim to encode the residual between the input and the information encoded by the previously updated latent variables. This can reduce the interference between the latent variables and encourage a disentangled latent representation. Moreover, the iterative encoding of the residual can enable high-order details of the input data to be encoded, which can lead to better reconstruction quality.

[0042] The final output of Res-VAE By and composing. The information bottleneck of a given VAE, relaxes the reconstruction constraint on the network. Such relaxation allows the network to better match the posterior with the prior of an isotropic unit Gaussian. This decomposition of the reconstruction is related to the problem representation in InfoGAN, where the input noise vector is partitioned into an incompressible noise source and a significantly structured semantic feature of the data distribution. The former corresponds to and the latter corresponds to In β -VAE, a similar idea is discussed, where latent factors are classified into disentangling and entangling generators. In this case, let denote the disentangling generators, and denote the entangling counterparts, so The data generation process can be formulated as . and are jointly learned using the objective in Equation 2. In Res-VAE, can be seen as a reconstruction from the entangling generators w , thus separating them from the disentangling generators. The difference between Res-VAE and the two above-mentioned representations is that, in the Res-VAE encoder / decoder network, among other things, it focuses on learning the disentangling factors. In contrast, the other approaches use the encoder / decoder network to learn both the entangling and disentangling factors. This difference can allow Res-VAE to reduce the trade-off between reconstruction and disentangling.

[0043] A closer look at the Res-VAE reconstruction shows that it can be . For L 2 the norm reconstruction loss, minimizing is equivalent to minimizing the cumulative decoder output from step 1 to as .

[0044] Each can be seen as an image transformation corresponding to the latent variable added at step i . This can minimize the impact of the added latent variable, thus seeking the minimum number of latent variables to represent the data. Together with the KL loss term, this representation encourages learning independent data generating factors. It should be noted that directly using the equivalent objective can lead to a solution where the network learns to output zero for all steps. In Res-VAE, this can happen rarely due to the presence of the residual term.

[0045] Res-VAE can utilize d forward passes instead of the standard forward pass of a VAE. However, in addition to improved state-of-the-art performance, it eliminates the hyperparameter tuning associated with other existing systems. Hyperparameter tuning can show to be critical to state-of-the-art performance. It can be a difficult and time-consuming process, especially for unlabelled data, due to: large hyperparameter search space for continuous values; lack of evaluation metrics, typically evaluated by human visual inspection. Thus, the added computational complexity by Res-VAE is justified compared to the cost of hyperparameter tuning. Furthermore, we will show that each of the d forward passes in Res-VAE corresponds to a disentangled generating factor. Visualization of these intermediate steps provides a new way of understanding the results.

[0046] Figure 2 Flowchart representing the Res-VAE mechanism. The residual learning mechanism can utilize d forward passes instead of a single forward pass. In the first pass and at step 201, the system can receive input data x . The input data x can be an image, a video, or some other type of object identification data.

[0047] At step 203, the system can pass the input data through the encoder and compute the parameterized posterior, which can be computed with the first p pass. At step 205, the system can sample the first latent variable and set the remaining latent variables to zero. Thus, instead of sampling all latent variables, only one latent variable is sampled. The modified latent variable can then be used to generate a reconstruction with the decoder.

[0048] At step 207, the system can then generate a reconstruction with the decoder. In step 207, the modified latent variable can be used to generate a reconstruction. At step 209, the difference between the reconstruction and the input data can be taken as input to the second pass. The second pass to the encoder generates a corresponding residual. The system can then sample the second latent variable with the corresponding residual and resample the first latent variable with the parameters learned in the first pass, all while setting the remaining latent variables to zero. The modified latent variables can then be used to generate a reconstruction and compute a corresponding residual. At step 211, the system can continue d times until all latent variables are sampled. At the p pass, the latent variables can be sampled from a distribution learned from the corresponding residual (e.g., table in Algorithm 1 for the pseudocode of the residual learning mechanism).

[0049] Figure 3 A block diagram 300 of a residual variational autoencoder (VAE) and residual learning software application is disclosed. Input 301 can be fed into an encoder 303 at a first pass 302. At Figure 3 In the block diagram 300 of FIG. 1, there can also be multiple passes, including a second pass 304, a third pass 306, and an Nth pass 308. Latent variables can be updated sequentially to learn a residual between input data and a reconstruction from a previously updated latent variable (e.g., 305, 309, 311, 313) during each forward pass (e.g., 302, 304, 306). The network can output a reconstruction Res-VAE can include an encoder 303 that can be and a decoder 307 that can be .

[0050] The decoder output and their corresponding residuals at each internal step can be output. The results can show that the decoder output and residuals can work together to capture learned disentangling factors at each step. For example, the factors in each step can be X-Position , Y-Position , Size , Rotation + Shape and Shape respectively. In one example, the X-positions of a reconstructed image are generated during the first step. In step two, both X-positions and Y-positions can be generated. The process can continue, and at each step, the output of the decoder 307 and residuals transform the image according to learned latent encodings.

[0051] The mechanism can decompose input data in a linear fashion as:

[0052] ,

[0053] where is a reconstruction from the previous latent variables, and represents a residual. This notation can encourage latent representations to match the linear decomposition assumption.

[0054] The “residual_learning” mechanism can be implemented by a enc ( ) and a decoder dec ( ) with the same (or similar) encoder dA forward pass consists of. Latent variables (e.g., 305, 309, 311, 313) can be sampled sequentially from the residual encoding between input data and cumulative reconstruction. At each step, the added latent variable can follow a distribution learned from the current residual, while previous latent variables can follow the same distribution learned from their corresponding residuals.

[0055] Figure 4 Examples of latent traversals across each latent dimension for an example dataset are disclosed. In cases where the interpretations of the change factors correspond to each latent dimension, the latent traversals across each latent dimension, where d is set to 10. The latent traversals across each latent dimension, where for the 2D shape dataset, d is set to 10. The internal process of reconstruction at each step in the Res-VAE method, i.e., the last column (a). Res-VAE is able to learn interpretable change factors, such as background, facial and hair characteristics, skin color, etc. In comparison to β- VAE and Factor VAE, the system can observe more background color tones, light source directions, etc. Careful tuning of the parameters can potentially reveal more latent variables. However, Res-VAE can eliminate the hyperparameter tuning, which can be a critical step for state-of-the-art performance. For example, Figure 4 Examples at each latent dimension for different attributes are shown, including background brightness, hair color, hair length, hair volume, age, azimuth, skin tone, hair direction, hair front height, background redness, etc. Of course, other attributes can be identified.

[0056] A possible advantage of Res-VAE is that it reduces the solution space and improves training stability. Res-VAE can have better performance and smaller variance, which means it can be less sensitive to random seeds. We also check whether using Equation 2 in conjunction with the “residual learning” mechanism β -VAE objective will improve performance, called Res- β -VAE. If so, the proposed “residual learning” mechanism will benefit from augmented objectives to achieve better performance. It should be noted that Res-VAE is a special case of Res- β -VAE, where β = 1. The best entanglement fraction is obtained by Res-VAE, and a higher β value does not help improve performance. These results validate the effectiveness of Res-VAE in addressing the trade-off between disentanglement and reconstruction.

[0057] The rearrangement of components in Res-VAE reveals that there can be similarities to ResNets. ResNets achieve state-of-the-art performance in various computer vision benchmarks. The key idea of ResNets is to utilize learning residual representations ( , where denotes stacked nonlinear layers) instead of learning a direct mapping between input and output ( ). This representation reduces the loss of important information when propagating through the network. Furthermore, it has been shown that learning residual mappings is easier compared to learning direct mappings. Another explanation for the effectiveness of ResNets is that they are equivalent to an ensemble of shallow networks.

[0058] In one example, a real image dataset does not have a ground truth generating factor. It can contain ten thousand different celebrity identities, each with twenty images. Each image is annotated with forty facial attributes, such as: young, pale skin, bangs, straight hair, no mustache, etc. In such an example, cropped facial images can be used as input data.

[0059] The metric can be computed by selecting latent factors k ; generating new data k with factors y fixed and other factors varying randomly; computing the mean of ; standardizing each dimension by its empirical standard deviation over all data or a sufficiently large subset (e.g., a threshold fraction, e.g., greater than 80%); constructing a majority vote classifier with the input having the index of the dimension with the lowest variance and the output of factors k . The classifier accuracy can be used as the evaluation metric.

[0060] In general, there can be three criteria for an unentangled representation, namely unentanglement, completeness, and informativeness. Unentanglement can be the extent to which the learned representation disentangles the underlying generating factors; completeness is the extent to which a latent representation captures the generating factors; and finally, informativeness is the amount of information about the generating factors captured by the latent representation. Unentanglement and completeness can be perceived by visualizing the rows and columns of the Sinton graph; and informativeness is computed based on the mapping error between the learned latent representation and the ground truth factors.

[0061] Accordingly, in an unsupervised setting, disentangled representation learning can be performed using Res-VAE. The "residual learning" mechanism in the training regime, as opposed to the augmentation objective, can allow a trade-off between disentanglement and reconstruction in VAE-based schemes. Res-VAE defines an explicit correlation structure among latent variables and decomposes the reconstruction into the decoder output and a residual term. Res-VAE can be applied directly to any VAE architecture without additional hyperparameter tuning. The step-by-step procedure of Res-VAE can visualize the results and understand the internal process of learning disentangled factors.

[0062] Embodiments of the present disclosure are described herein. It is to be understood, however, that the disclosed embodiments are merely examples and other embodiments can take various and alternative forms. The figures are not necessarily to scale; some features can be exaggerated or minimized for the purpose of clarity and illustration. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the embodiments. As those skilled in the art will appreciate, the various features illustrated and described in connection with any one figure can be combined with features illustrated and described in connection with one or more other figures, to produce embodiments that are not explicitly illustrated or described. The combinations of features illustrated and described in connection with the figures are not to be construed as limitations on the scope of the disclosure. It will be apparent to those skilled in the art that various modifications and variations can be made in the present disclosure without departing from the spirit or scope of the disclosure.

[0063] The processes, methods, or algorithms disclosed herein can be deliverable to / implemented by a processing device, controller, or computer, which can include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored as data and instructions on a non-transitory storage medium included in or accessible to the controller or computer. The processes, methods, or algorithms can also be embodied in a software executable object. Alternatively, the processes, methods, or algorithms can be embodied in whole or in part using any suitable hardware component, such as an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), state machine, controller, or other hardware components or devices, or a combination of hardware, software and firmware components, and can implement one or more electrical components under the control of software and / or firmware applied to the hardware components. For example, the software executable object can be implemented in a virtual machine, or the software executable object can be implemented in a cloud computing environment.

[0064] While example embodiments have been described above, these embodiments are not intended to describe all possible forms of the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the disclosure. As previously described, features of various embodiments can be combined to form further embodiments of the present disclosure that can not be expressly described or illustrated. While various embodiments can have been described as providing advantages or being preferred over other embodiments or prior art implementations, this description is not intended to limit the scope of the claims to these particular embodiments. It is intended that the description serve as an enabling description of various embodiments of the present disclosure, and as such, various changes can be made in the frame of one or more features, but equivalent elements can be substituted for other elements in order to achieve the same property, and equivalents to each feature can be substituted in order to achieve the same or similar result in the eventual retention of such changes as can be necessary. As such, it is intended that the disclosure not be limited to the explicitly described embodiments, but rather, the full scope of the disclosure will be determined by the scope of the claims, together with any equivalents thereon.

Claims

1. A system comprising: a camera configured to obtain image information from an object; and a processor in communication with the camera and programmed to: receive input data comprising the image information, encode the input via an encoder, during a first sequence, obtain a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, output a final reconstruction of the input data with a plurality of residuals from a plurality of sequences, and output a final latent representation comparing the first latent variable from at least the first sequence to a second latent variable in a second sequence.

2. The system of claim 1, wherein, the processor is configured to update only one latent variable during each forward pass sequence.

3. The system of claim 1, wherein the plurality of sequences comprises a plurality of latent variables utilized.

4. The system of claim 1, wherein the encoder is a neural network encoder.

5. The system of claim 1, wherein the input data comprises video information obtained from the camera.

6. An apparatus comprising: a processor programmed to: receive input data, encode the input via an encoder, during a first sequence, obtain a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, output a final reconstruction of the input data with a decoder and a plurality of residuals from a plurality of sequences, and update the first latent variable after the first sequence and the second latent variable after a second comparison with at least the second latent variable.

7. The apparatus of claim 6, wherein the processor is further programmed to obtain a residual between the input data and the successive reconstructions from the previously updated latent variable.

8. The apparatus of claim 6, wherein the processor is further programmed to compare the residual between the first latent variable and the second latent variable in the first sequence and compare the residual between the first latent variable and the second latent variable in the second sequence.

9. The apparatus of claim 6, wherein the encoder is a neural network encoder.

10. The apparatus of claim 8, wherein, the processor is further programmed to update the latent variable during each forward pass during a plurality of forward pass sequences.

11. A computer program product storing instructions that, when executed by a computer, cause the computer to: receive input data, encode the input via an encoder, during a first sequence, obtain a first latent variable defining a property of the input data, generate successive reconstructions of the input data with a decoder and at least the first latent variable, obtain a residual between the input data and the reconstructions with a comparison of at least the first latent variable, output a final reconstruction of the input data with a plurality of residuals from a plurality of sequences, and output a final latent representation comparing the first latent variable from at least the first sequence to a second latent variable in a second sequence.

12. The computer program product of claim 11, wherein the input comprises an image received from a camera in communication with the computer.

13. The computer program product of claim 11, wherein the encoder comprises a neural network.

14. The computer program product of claim 11, wherein, The computer program product further comprises instructions that cause the computer to decompose the input data by a residual between the input data and a reconstruction of the latent variable for a previous update.

15. The computer program product of claim 11, wherein, The computer program product further comprises instructions to update only one latent variable during each sequence of forward passes.

16. The computer program product of claim 11, wherein, The computer program product further comprises instructions to, during a second sequence, obtain a second latent variable, generate a second successive reconstruction of the input data with the decoder and the at least second latent variable, obtain a second residual between the input data and the reconstruction with the at least second latent variable.

17. The computer program product of claim 11, wherein the input data comprises a dataset of multiple images.

Images