Controllable potential space discovery using multi-step inverse model

Abstract

Devices, systems, and methods for determining a minimally controllable potential state and operating a model trained to implement the minimally controllable potential state are generally discussed herein. A method may include receiving a first observation and a second observation of a time series of observations generated by a sensor, the second observation being separated from the first observation by more than one other observation, the first observation being associated with a first action performed by an agent in response to the first observation; encoding, by an encoder, the first and second observations into respective first and second hidden state representations; combining the first hidden state representation and the second hidden state representation to obtain a combined representation; operating the combined representation by an action predictor to obtain a second action; and changing weights of the action predictor and the encoder based on a difference between the first action and the second action.

Description

Background Art

[0001] One-step inverse models are very popular, including recent work by OpenAI called video pre-training (VPT). VPT fails to capture a complete, controllable latent state. Models that predict the entire observation space, such as Google LLC's Dreamer model, capture information including control, but also information that is not relevant to control. To the best of the inventors' knowledge, there is currently no known technique that provides a general theoretical guarantee for discovering the minimal controllable latent state. Summary of the invention

[0002] An apparatus, system, method, and computer-readable medium configured to determine a minimum controllable potential state and operate a model trained to achieve the minimum controllable potential state. The minimum controllable potential state can be learned using a multi-step inverse model. The multi-step inverse model can include an encoder that encodes a series of observations. The encoded observations can be provided to an action predictor such that consecutive observations are separated by a specified number k of observations greater than or equal to 2. The encoded observations can be focused before being provided to the action predictor. Focusing the encoded observations can reduce the encoded observations to a minimum controllable potential state. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A diagram of an embodiment of a system for determining a minimal controllable latent space representation of an agent is shown by way of example.

[0004] Figure 2 A diagram showing an embodiment of a system is shown by way of example, the system comprising a Figure 1 A trained model of the system.

[0005] Figure 3 A diagram of an embodiment of a method for determining a minimum controllable latent state and / or operating a model trained to achieve the minimum controllable latent state is shown by way of example.

[0006] Figure 4 A diagram representing potential states of a robot arm that may be controlled is shown by way of example.

[0007] Figure 5 Shown are images produced by a network trained using AC-State.

[0008] Figure 6 By way of example, plots of percentage error versus number of training epochs for various networks trained in various ways, including autoencoders, single-step inverse models, and networks trained using the AC states of an embodiment are shown.

[0009] Figure 7A diagram showing an embodiment of another experiment for learning controllable latent states of a target is shown by way of example.

[0010] Figure 8 State parsimony score graphs for various machine learning (ML) models are shown by way of example.

[0011] Fig. 9 Plots of multiple accuracy metrics for various ML models are shown by way of example.

[0012] Fig.10 is a block diagram of an example of an environment including a system for neural network training.

[0013] Fig.11 A block diagram of an embodiment of a machine (eg, a computer system) that implements one or more embodiments is shown by way of example. DETAILED DESCRIPTION

[0014] In the following description, reference is made to the accompanying drawings that form a part of the description, and wherein by the mode of specific embodiments that can be put into practice, it is shown. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that other embodiments can be utilized, and structural, logical and / or electrical changes can be made without departing from the scope of the embodiments. Therefore, the following description of the embodiments should not be considered to have a limiting meaning, and the scope of the embodiments is limited by the appended claims.

[0015] Someone walking down a city street trying to model all aspects of the world will quickly be overwhelmed by the sheer number of stores, cars, and people entering and exiting the field of vision following their own complex and elusive dynamics. Exploration and navigation in such environments are daily tasks that do not require a great deal of mental resources. Is it possible to transform this torrent of perceptual information into a minimal latent state that is necessary and sufficient for an agent to successfully act in the world? This problem is concretized, and embodiments propose an Agent-Controllable State (AC-State) discovery algorithm that has theoretical guarantees and is proven in practice to be able to discover the minimal controllable latent state that contains all the information necessary to control the agent, while completely discarding all irrelevant information. Embodiments include a multi-step inverse model with an information bottleneck (predicting actions from distant observations (observations more than one time step)). AC-State allows localization, exploration, and navigation without rewards or demonstrations. Discovery of controllable latent states has been implemented in three domains: localizing a robotic arm in the presence of disturbances (e.g., changing lighting conditions and background), exploring a maze with other agents, and navigating in a Matterport house simulator.

[0016] Embodiments include a system for learning the dynamics and states related to control agents in the world while discarding phenomena unrelated to the control agents. The system is completely unsupervised and includes recording or accessing a time observation stream (e.g., video, audio, image or text log) and the actions corresponding to the time stream taken by the agent for the corresponding observations of the time observation stream. The model trained according to the embodiment predicts actions from the current observation and one or more future observations (observations greater than one time step in the future) while keeping the hidden state of the system to a minimum. The potential state found (sometimes referred to as a hidden state) can be used to visualize the partial observations interacting with the agent while removing irrelevant partial observations (such as background television (TV), background noise, irrelevant text or graphics, or changing lighting conditions, or a combination thereof, etc.). The actions of the agent can be taken randomly, by a predefined expert agent, or by planning in a learned dynamic model. The latter can be used to achieve efficient exploration in a noisy environment. The learned dynamic model can also realize navigation and control of the agent.

[0017] Several scientific fields, from classical mechanics, chemistry, and evolution to subfields of medicine, have concise representations for encapsulating how systems (or agents) interact with the world. For example, consider a world consisting of a robotic arm agent that manipulates a target using a camera-based sensor. The high-dimensional camera observations of the agent contain intertwined information about the agent's control and many interferences from external sources such as lighting conditions. Concise representations improve the efficiency of many applications by capturing information such as the joint angles between the arm's links and excluding visual details that are irrelevant to the agent's dynamics. A controllable latent state is defined as a concise representation that includes only information that can be changed by the agent (such as a target on a table that can be manipulated by the arm) or information that affects the agent's actions (e.g., an obstacle blocking the movement of a robotic arm). How can a concise representation be discovered from observations that captures only the underlying dynamics of the agent-world interaction? The AC-State technique of an embodiment provably guarantees the discovery of a controllable latent state from observations, excluding all aspects of the observation that are not necessary for control.

[0018] AC-State learns an encoder f that maps a given observation x to a corresponding controllable latent state f(x). This is achieved by optimizing new objectives using expressive model classes such as deep neural networks (DNNs). Controllable latent states enable agents to efficiently and autonomously localize, explore, and navigate within the world. As an example, a household robot vacuum cleaner must cover the entire area of ​​the house to provide effective cleaning. This can be done quickly using AC-State. For example, a robot can use the latent state encoder learned by AC-State to locate within a controllable latent space, plan to reach a specific room and cover the floor there, and then execute the plan in the real world. Because the controllable latent state excludes irrelevant noise in the house, such as changes in light conditions during the day, each step of the process is efficient.

[0019] Learning controllable latent states can be more formally characterized by considering a discrete-time dynamical system where observations x1,x2,x3,…,x T Evolving according to the observed dynamics, we write it as the conditional probability distribution: T(x t+1 |x t ,a t ). Select a t As x t The choice is a policy that can be learned or manually designed. The latent dynamics can then be decomposed into T(s t+1 |s t ,a t )T(e t+1 |e t ), where s is the controllable latent state and e is exogenous information in the observation space that is irrelevant to control. Successful discovery of the controllable latent state only requires learning T(s t+1 |s t ,a t ) and how to encode s without learning anything about T(e t+1 |e t ) or knowledge of how to encode e.

[0020] Deep learning architectures can be optimized for a wide range of differentiable objective functions. An unanswered question is: what is the goal of a provably learnable controllable latent state that is compatible with deep learning? The problem is to find a concise representation that is sufficient to control a dynamic system while discarding irrelevant details given observations from rich sensors (such as high-resolution video). Methods such as optimal state estimation, system identification, and simultaneous localization and mapping achieve concise state estimation for control, but require more domain expertise and design than desired. Previous learning-based methods fail to capture fully controllable latent states or fail to exclude all irrelevant information. Reinforcement learning methods that capture latent states from rich sensors using autoencoders or contrastive learning often capture noisy components1. In contrast, inverse model-based methods may not capture fully controllable latent dynamics, and combining them with autoencoders inherits the shortcomings of this approach. Reward-based dual simulation can filter irrelevant information from the latent state, but relies on access to a reward signal. Deep reinforcement learning based on reward optimization often struggles with sample complexity when rewards are sparse, and fails completely when rewards are absent.

[0021] A controllable latent state should preserve information about the interactions between states and actions while discarding irrelevant details. The proposed objective (AC-State) achieves this by generalizing the one-step inverse dynamics to multiple steps with an information bottleneck:

[0022] The parameters of the encoder model are optimized as It maps from n-dimensional continuous observations to a finite number of latent states with an output range of size Range(f). Represents all mappings that can be realized by the model. Multi-step inverse objective From taking action a t The previous observations and the observations collected in the next k steps are used to predict action a t The action step t may be sampled over all T collected samples (eg, uniformly), while the prediction horizon k may be sampled from 1 to K (eg, uniformly).

[0023] Controllable latent states are guaranteed to be used for multi-step inverse objectives one of the optimal solutions, while other solutions may not be able to remove irrelevant information. In addition, controllable latent states are only achievable by finding the lowest capacity f that minimizes the objective. Embodiments may use two mechanisms to limit the information capacity in f. The latent states at the end of the network can be passed through a Gaussian variational information bottleneck. The Gaussian variational information bottleneck reduces the mutual information between x and the representation. Vector quantization can be applied to produce discrete latent states. Although either of these can be used alone, adding the Gaussian mutual information objective simplifies the discovery of concise discrete representations. In addition to optimizing the objective and limiting the capacity, the actions taken by the agent are also important to the success of AC-State, as they must achieve high coverage of the controllable state space and be independent of exogenous noise. This is achieved through random strategies or dependence on and achieve high coverage strategies to satisfy the AC-State objective, which enjoys provable asymptotic success in discovering controllable latent states.

[0024] Intuitively, the AC-State objective encourages latent states to retain information about the long-term effects of actions, which requires storing all information about how actions affect the world. At the same time, the AC-State objective never needs to predict information about the observations themselves, so it has no value in representing a world that is independent of the agent.

[0025] Learning latent states in interactive environments is a mature research area with rich contributions. We provide some research lines and how they fail to achieve guaranteed discovery of controllable latent states. These contributions are divided into three broad areas based on what they predict: latent state prediction, observation prediction, observation relationship prediction, and action prediction.

[0026] Predicting latent states: In reinforcement learning, learning latent state representations from high-dimensional observations involves maximizing the reward to solve the task. Here, the model predicts latent states such that the pre-trained representation can solve the task. Deep dual-simulation methods learn state representations for control tasks, ignoring details that are not relevant to the task. Autoencoders or dynamic models trained with reconstruction losses learn low-dimensional state representations to capture information relevant to solving the task.

[0027] Although learned latent state representations are shown to be useful for solving tasks, there is no guarantee that such methods can fully recover the underlying controllable latent state. The learned representation captures both controllable and exogenous parts of the latent state, even though the pre-trained representation can be used to solve the task. In contrast, AC-State fully recovers the controllable state with theoretical guarantees, while experimental results demonstrate that the underlying controllable latent structure is recovered, independent of the exogenous parts of the state.

[0028] Predicting Observations: Prior work has learned generative models or autoencoders to predict future observations for learning latent representations. This is primarily used for exploration. Intrinsic reward signals for exploration in complex domains are implemented by training an inverse dynamics model to predict actions while training a dynamics model to predict future observations. The idea of ​​predicting observations is often referred to as an intrinsic motivation that guides the agent to explore unseen regions in state space. Other work uses autoencoders to estimate future observations in feature space for exploration, although such models fail in the presence of exogenous observations. Dynamics models learn to predict a distribution over future observation predictions, but are typically used to plan action sequences rather than recovering latent states or for exploration purposes. In theory, existing work has learned representations based on predicted future observations, making the learned representations useful for exploration; however, one-step predictive dynamics models have been shown to be mandatory to capture exogenous noise.

[0029] Instead of predicting future observations, AC-State predicts actions based on future observations, and we show that a multi-step inverse dynamics model that predicts actions can fully recover the controllable latent state independent of exogenous noise. In the presence of exogenous noise, methods based on predictive future observations tend to predict both the controllable and exogenous parts of the state space and are not guaranteed to recover the latent structure.

[0030] Predicting Relationships Between Observations: Existing work attempts to learn latent state representations both theoretically and empirically by learning to predict the relationship between two consecutive observations. By learning representations that exploit mutual information-based objectives (information gain based on the current state and actions with future states), previous work has attempted to learn controllable states in the presence of exogenous noise. However, unlike AC-State, they learn latent states that depend on exogenous noise, even though the learned representations are useful for solving complex tasks. In theory, some use contrastive loss-based objectives to provably learn latent state representations, which may be useful for hard exploration tasks. However, contrastive loss-based representations are still prone to exogenous noise, while AC-State exploits an exogenous-free random expansion strategy to provably and experimentally recover fully controllable latent states through a multi-step inverse dynamics model.

[0031] Predicting Actions: AC-State aims to recover a controllable latent state by training a multi-step inverse dynamics model in the presence of exogenous noise. While existing work has explored similar goals, both for exploration and for learning state representations, existing work cannot recover the latent state to perfect accuracy in the presence of exogenous noise. The idea of ​​using a simple one-step inverse dynamics model has been explored in the past, however, the one-step inverse model has counterexamples showing that it cannot capture a fully controllable latent state.

[0032] Intuitively, the one-step inverse model is underconstrained and may therefore incorrectly merge different states in a Markov decision process (MDP) that are far apart but have similar local structure. As a simple example, suppose there is a chain of states: s1, s2, s3, s4, s5, s6 and a=0 moves earlier in the chain and a=1 moves later in the chain. Assume that s1, s3, s4, s6 are encoded as different latent states and s2, s5 are merged into the same latent state, which we can call s * . Contains * Examples of inverse models are: (s1,s*,1), (s * ,s1,0),(s * ,s3,1),(s3,s * ,0),(s4,s * ,1),(s * ,s4,0),(s * ,s6,1),(s6,s * ,0). Because all these examples have different inputs, the one-step inverse model still has zero error despite the incorrect merging of states s2 and s5.

[0033] Empowerment-based objectives focus on the idea that the agent should try to find states that empower it to have the largest number of easily reachable possible states. For example, in a maze with two rooms, the most empowered state is the doorway, since it makes it easy to reach any room. Specific instances of the empowerment objective might involve training a model to predict a distribution over actions from observations (single-step or multi-step inverse models), but lacking the information bottleneck term and the requirement for an expansion strategy that is independent of external factors. The analysis and theory in this work focus on action prediction as a specific method for measuring emptying, rather than as a way to guarantee the discovery of a state with minimal controllable delay and to ignore extraneous noise.

[0034] In contrast, AC-State uses a simple multi-step inverse dynamics model with a randomized rollout strategy and provably guarantees perfect recovery of only the controllable portion of the state space in the presence of exogenous noise (either from other agents acting randomly in the environment or from the presence of background interferers). The simplicity of AC-State using a simple inverse dynamics model to learn the latent state has not been fully exploited by existing research using dynamics models.

[0035] Provably Reinforcement Learning and Control in the Presence of Exogenous Noise. The reinforcement learning problem in the presence of exogenous and irrelevant information has been analyzed. For example, the Ex-BMDP model was designed as a provably efficient algorithm for learning state representations. However, the algorithm only succeeds in the setting when the initial controllable state is initialized deterministically. This strict assumption makes the algorithm impractical in many interesting situations. AC-State removes the deterministic assumption of the initial latent state. In fact, as shown theoretically, AC-State can be applied in the you-only-leave-once setting when the agent visits a single trajectory.

[0036] Other provable RL algorithms have been designed that are designed to learn efficiently the underlying dynamics in the presence of different assumptions about exogenous noise. However, these works focus on the statistical aspects of the problem; how to scale these methods to complex environments and combinatorial function approximations is currently unknown and appears challenging. Unlike the aforementioned work, the embodiments focus on the representation learning problem. Our goal is to design a practical and guaranteed method by which we can learn controllable representations using complex function approximators such as deep neural networks. AC-State is the first algorithm to achieve this task.

[0037] Figure 1 An embodiment of a system 100 for determining a minimal controllable latent space representation of an agent 102 is shown by way of example. The minimal controllable latent space is jointly learned by an encoder 108 and an action predictor 112. The sensors 104 provide corresponding observations 114 to a trainer 106 and an agent encoder 136. The agent encoder 136 may be operable to encode the observations 114, thereby obtaining encoded observations 138. Alternatively, the observations 114 may be retrieved from a data set storing the observations 114 (e.g., based on a heuristic or randomly).

[0038] The agent 102 performs corresponding actions 116 based on one or more encoded observations 138 of the observations 114. The actions 116 of the agent 102 are recorded and provided to the trainer 106 or observed and recorded by the trainer 106. Each of the observations 114 and actions 116 can be associated with a corresponding timestamp. The timestamp can indicate the date and time when the associated observation 114 was generated. The timestamp can indicate the date and time when the associated action was performed. A series of observations 114 ordered in time is sometimes referred to as a "time series of observations." A series of actions 116 ordered in time is sometimes referred to as a "time series of actions."

[0039] The time series of observations and the time series of actions may be aligned with respect to time such that an observation 114 is received by the agent 102 and the corresponding action of actions 116 is the next action performed by the agent 102 immediately after receiving the corresponding observation 114 .

[0040] Agent 102 may include a device, system, circuit, electrical tool, mechanical tool, chemical tool, or a combination thereof. Agent 102 is operable to receive observations 114 and perform actions 116. Observations 114 are outputs of sensors 104. Sensors 104 are transducers that convert electromagnetic, mechanical, or chemical inputs into electrical signals to indicate conditions monitored by sensors 104. The conditions may include light, salinity, pH, force, pressure, gas (e.g., oxygen, carbon dioxide, carbon monoxide, steam, etc.), conductivity, proximity to another target, movement (e.g., speed, direction, etc. via an accelerometer), orientation (e.g., via a tilt sensor), moisture content (e.g., via a humidity sensor), human touch, altitude, geographic location (e.g., via a global positioning system (GPS), etc.), and many other possible conditions. Actions 116 may include movement, direction of movement, amount of movement, inhibiting action, capturing images, starting or stopping device operation, activating algorithms, mechanical movement, chemical release or capture, changing display, and others. In some embodiments, the sensor 104 and the agent 102 are integrated into the same device, and in other embodiments, the sensor 104 and the agent 102 are part of different devices.

[0041] The trainer 106 is a computing device that provides a series of selected corresponding observations 118 from the observations 114 to the encoder 108. The trainer 106 controls the training process of the joint training of the encoder 108 and the action predictor 112. The trainer 106 provides the observation 118 at index J in the time series of observations. J is an integer in the range (0, maximum observation index].

[0042] The trainer 106 provides an observation at index J (observation 118) in a first training iteration, and then the trainer provides an observation a number of indices before observation 118 (observation at index J+K) in the immediately next iteration. K is an integer greater than 1 and less than the maximum observation index. K is the number of indices of the observation that immediately precede the previous observation 118 in the time series of observations. The trainer 106 thus jointly trains the encoder 108 and the action predictor 112 in a multi-step manner (using observations that are multiple steps (e.g., more than one step) apart in the time series of observations). An upper limit on K can be defined. In some examples, the upper limit can be defined as the maximum path length of the Markov decision process of the agent's behavior.

[0043] The encoder 108 transforms the observations 118 into a lower dimensional space. The output of the encoder 108 is a hidden state 124, which is a vector of fixed length. The encoder 108 can be implemented using a convolutional neural network (CNN), a transformer, or another neural network (NN) construction.

[0044] The hidden state 124 may be received by a hidden state focusing operator 132. The hidden state focusing operator 132 may perform operations such as a Gaussian variational information bottleneck, vector quantization, another discretization of the hidden state 124, another technique to reduce the mutual information between the observations and the hidden state 124, combinations thereof, etc. The focused hidden state 134 includes the hidden state 124 with uncontrollable factors removed from the hidden state 124. Thus, the focused hidden state 134 is a minimal or near-minimal representation of information for the action predictor 112 to determine the action 126.

[0045] The action predictor 112 may operate on the focused hidden state 134. The action predictor 112 may include any model capable of receiving and operating on the focused hidden state 134 and generating the action 126. The action predictor 112 may include a decoder NN or other model capable of decoding the action 126 from the focused hidden state 134.

[0046] The loss operator 110 may receive an action 128 associated with the observation 118 and an action 126 determined by the action predictor 112. The loss operator 110 may generate an error 130 based on an objective function. The error 130 may be used to adjust the weights, parameters, or a combination thereof of the action predictor 112, the encoder 108, etc. The adjustment may be implemented using back propagation. The objective function may be expressed as: where t is a time or index, k is an integer greater than one, x is an observation, and a is an action.

[0047] The operation of the hidden state focusing operator 132 can be operated to make the focused hidden state 134 as small, coarse or contain as little information as possible. This can be expressed as G:

[0048] Further details regarding training and operating the encoder 108 and motion predictor 112 and applications are provided elsewhere.

[0049] The result of training with AC-State can be represented by a discrete graph, where there is a node for each controllable latent state with edges representing actions leading to the state-action node and weighted result edges from the state-action node back to the controllable latent state node. T(s) is estimated by using the counts t+1 |st , a t ), the probability of transitioning from one node to another can be determined in this controllable latent space. After this distribution of controllable latent states and associated transitions is estimated, it is straightforward to measure how correct and parsimonious these dynamics are, as long as the true controllable latent states are available (which is the case in all experiments discussed in this paper). Errors (e.g., L1, L2, or other errors) can be determined from this dynamic distribution and the ratio of the number of learned latent states to the number of true controllable states, which we refer to as state parsimony. Experiments discussed in more detail elsewhere explore three domains and demonstrate the unique capabilities of AC-State: (i) AC-State learns the controllable latent states of a robot from high-resolution videos of the robot that contain rich temporal context structure (a TV playing a video, flashing lights, a flapping bird, and even a person) that dominates the information in the observed dynamics; and (ii) multiple mazes and functionally identical agents, only one of which is controlled. AC-State is able to solve the hard maze exploration problem by learning only the controlled agent and ignoring the others. (iii) AC-State learns a controllable latent state in a house navigation environment, where observations are high-resolution images and the vertical position of the camera oscillates randomly, showing that AC-State is invariant to exogenous viewpoint noise that fundamentally changes the observations.

[0050] Figure 2 A diagram showing an embodiment of a system 200 is shown by way of example, the system comprising a Figure 1 The trained model of the system 100 of FIG. The trained model includes the encoder 108, the hidden state focusing operator 132, and the trained action predictor 112. The system 200 as shown includes the sensor 104, which generates another observation 220 after deployment. The encoder 108 and the action predictor 112 are jointly trained to produce an action 222 based on the observation 220. The action 222 is reliably the action 116 that the agent 102 would take in response to the observation 114. Therefore, the encoder 108, the hidden state focusing operator 132, and the action predictor 112 can be used to replicate the action 116 of the agent 102. More details about possible actions, agents, and other details are provided elsewhere.

[0051] Figure 3A diagram of an embodiment of a method 300 for determining a minimum controllable latent state and / or operating a model trained to achieve a minimum controllable latent state is shown by way of example. The method 300 as shown includes receiving a first observation in a time series of observations generated by a sensor and a second observation in a time series of observations, the second observation being separated from the first observation in the time series of observations, the first observation being associated with a first action performed by an agent in response to the first observation at operation 330; encoding the first observation and the second observation into corresponding first hidden state representations and second hidden state representations by a neural network (NN) encoder at operation 332; combining the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation at operation 334; operating the combined hidden state representation by an action predictor at operation 336 to obtain a second action; and changing the weights of the action predictor and the encoder based on the difference between the first action and the second action at operation 338.

[0052] The method 300 may also include changing the hidden state representation to remove information not relevant to control of the agent, resulting in a focused hidden state representation, and wherein the action predictor operates on the focused hidden state representation. The method 300 may also include, wherein changing the hidden state representation includes using a Gaussian variational information bottleneck, vector quantization, or a combination thereof. The method 300 may also include, wherein changing the hidden state representation includes using a Gaussian variational information bottleneck and vector quantization to generate the focused hidden state representation.

[0053] The method 300 may also include, wherein changing the weight includes determining the probability using an objective function based on the operation of the first action, the first observation, and the second observation. The method 300 may also include receiving a value k indicating a difference between a first index of a first observation and a second index of a second observation in the time series of observations. The method 300 may also include, between consecutive training iterations, changing k to a different integer. The method 300 may also include, wherein k varies between two and a maximum path length in a Markov decision process that models the behavior.

[0054] AC-State discovers controllable latent states of a real robot arm while ignoring distractors. Six hours of data (14,000 samples) are collected from the operation of the robot arm. In the data, the robot arm performs four high-level actions (move left, move right, move up, and move down). Photos of the robot arm are taken after each action is completed. The robot arm is arranged in the presence of many distractors, such as TV, flashing lights, color-changing lights, and people moving in the background. Some of these distractors (especially TV) have strong temporal correlations between adjacent time steps, which is often the case in real life.

[0055] Figure 4 A graph 400 representing controllable latent states of a robot arm is shown by way of example. The controllable latent states are the real positions of the robot (not used during training). The graph 400 includes nodes 440 representing the positions of the ends of the robot arm, and arrowed edges 442 represent possible transitions between nodes 440. The AC-State is evaluated by training a neural network to reconstruct the latent state x in images of the robot arm's motion f(x). The position of the robot arm is perfectly reconstructed using the learned latent space, while the interfering TV and color-changing lights appear completely blank as expected.

[0056] Figure 5 An image produced by a network trained using AC-State is shown. It can be seen that items that do not affect the state of the robot arm (items outlined in the thick solid line 550 on the left do not appear in the thick solid line 552 on the right), including the video and the target in motion, are not copied. Instead, the minimum potential state is retained, focusing on the position of the robot arm.

[0057] Figure 6 A graph 600 of percentage error versus number of training rounds is shown by way of example for various networks under various training methods, including an autoencoder (represented by line 660), a single-step inverse model (represented by line 662), and a network trained using an embodiment of AC-State (represented by line 664). The autoencoder is very inaccurate because it captures controllable latent states and uncontrollable states such as interfering noise (items in solid line 550). The inverse model is more accurate than the autoencoder, but still has the problem of capturing uncontrollable states. The AC-State network learns controllable latent states and can almost perfectly replicate the robot arm movements after a sufficient number of training rounds.

[0058] A robotic arm with 6 degrees of freedom is used, with 5 possible abstract actions: forward, backward, left, right, and stay. The robotic arm moves within 9 possible positions in a virtual 3×3 grid with walls between some cells. The center of each cell is equidistant from the adjacent cells. The end effector is kept at a constant height. Each cell center of mass is calculated, and the robot's joint space is transformed to a specific grid cell via standard inverse kinematics calculations. Still images are taken using two cameras. One camera faces the front of the robot, and the other camera is from above the robot down. When a command is received, assuming that no walls exist, the robot moves from one cell to the center of another cell. After each move, a still image (640×480) is taken using two cameras and appended together into one image (1280×480). During training, only front-facing, downsampled (256×256) images are used. Each move takes one second. After every 500 joint space moves, the robot is recalibrated to the grid to avoid position drift.

[0059] Data was collected for 6 hours of the robot arm moving under a uniform random strategy, resulting in a total of 14,000 samples. There were no events or state resets. In addition to the robot, several distractors were present in the image. The video was looped at 2× speed on a large monitor in high resolution (4K video). There were also four drinking bird toys, color-changing lights, and flashing running water lights. During the last half hour of image collection, distractors were moved and / or removed to simulate additional uncertainty in the environment.

[0060] By learning a small convolutional neural network, the latent state f(x t ) is mapped to the observation x t Estimates And by optimizing the mean square error reconstruction loss To learn visualization of latent states.

[0061] Figure 7 A diagram showing an embodiment of another experiment for learning controllable latent states of a target is shown by way of example. Figure 7 The example experiments in are multi-agent systems where only a single agent among the multiple agents has a controllable latent state, while the rest are uncontrollable exogenous agents controlled by a random policy. As discussed earlier, the autoencoder trained end-to-end with x as input captures both the controllable latent state and the interfering noise. In our experiments, AC-State removes the exogenous noise in the environment even when the noise has a rich and complex structure. Figure 7In the specific example of , the agent 770 steered in the lower right path 772 is the only controllable agent. The other agents 774 follow their own independent strategies to navigate their paths 776. The learned controllable latent states 778 perfectly replicate the paths and connections between them, while ignoring the uncontrollable and exogenous paths 776.

[0062] exist Figure 7 In the example of , nine (9) agents are shown, of which only one is controllable. However, embodiments are applicable to more or fewer agents, of which more or fewer are controllable. For 9 agents, each with c controllable states, the total size of the observation space is c 9 With 3,000 training samples, AC-State is able to almost perfectly discover the controllable latent states of agent 770 while completely ignoring the states of eight (8) uncontrollable exogenous agents.

[0063] Controllable latent states are useful when they allow more efficient exploration than is achievable under a random policy. To demonstrate this, we use Figure 7 A modified maze problem where the agent is given additional actions to reset to a fixed initial starting position. When one third of all actions cause a reset, the probability of taking a random action in N steps without resetting is (2 / 3) N The learned exploration strategy using AC-State successfully fully explores and learns the controllable latent states with 25,000 samples, while the random exploration strategy barely explores the first room with the same number of samples.

[0064] In the reset experiment, the agent can cause a reset to a fixed starting state if it chooses the reset action. In addition, a single perturbed maze is used as a source of exogenous noise. The goal of this experiment is to show that in the presence of a reset action, a random expansion strategy has difficulty obtaining complete coverage of the maze. To achieve sufficient coverage, the discovered controllable latent states can be used to learn a goal-seeking strategy that can be motivated to deterministically reach unseen regions of the state space. The counts of discrete latent states are used to construct a simple tabular Markov decision process (MDP), where planning is done using a Monte Carlo version of Djikstra's algorithm (to account for random transition dynamics) to reach the goal state. The sampling ratio of reachable goal states is proportional to 1 / count(s i ), making the rarely seen states most likely to be selected as the goal. Experimental results demonstrate that the goal-seeking strategy is able to perfectly cover the state space by exploring using the discovered latent states, while the random strategy fails to reach more than 25% of the state space in the presence of the reset action.

[0065] For comparison, a 2-layer feed-forward network (FFN) with 512 hidden units is used for the encoder network, followed by a vector quantization (VQ-VAE) bottleneck. The representation from the multi-step inverse model is discretized using a VQ-VAE bottleneck by adding a codebook of discrete learnable codes. To recover a controllable latent state from the maze for control while ignoring other exogenous mazes, an MLP mixer architecture with gated residual updates is adopted. Both the inverse model and the VQ-VAE bottleneck are updated using the Adam optimizer with the default learning rate of 0.0001 without weight decay. The agent can receive abstract observations or pixel-based observations of size 80×80×3. For the multi-maze experiment, the size of the observations received by the agent is 80×720×3 due to observations from 8 other exogenous agents. The agent's action space is 4, where the actions are randomly picked from a uniform policy. For the reset action setting, we use an additional 4 reset actions and uniformly pick the reset action that can reset it to a deterministic starting state.

[0066] To analyze the performance of the AC-State objective in a more realistic setting, AC-State is evaluated on Matterport, a navigation environment with high-resolution images taken from locations in a real house. Navigation through the house is learned from a dataset with 20,000 samples collected from an agent randomly moving through the house. In addition to the high degree of visual information in the input observations, the camera is moved up or down at each step as a controlled source of irrelevant information (exogenous noise). AC-State removes view noise from the encoded representation f(x) while still capturing realistic and controllable latent states ( Figure 8 and Fig. 9 ), while the other baselines capture both controllable latent states and exogenous noise.

[0067] The Matterport simulator contains an indoor house in which an agent can navigate. The house contains a finite number of viewpoints to which the agent can navigate. At each viewpoint, the agent can control its viewing angle (by rotating left or right by a certain angle) and its elevation angle: there are a total of 12 possible viewing angles and 3 possible elevation angles per viewpoint. The data is collected using a random unrolling strategy. At each step of the unrolling strategy, the agent navigates to a neighboring viewpoint. The viewpoint is randomly changed in some steps of the unrolling strategy in order to introduce exogenous information that the agent cannot control. A single long event of 20,000 state transitions is collected. The controllable latent state in this setting is the viewpoint information, while the exogenous information is information about the agent's elevation angle.

[0068] The model input is a panorama of the current viewpoint image for 12 possible views for each viewpoint. The AC-State model f is parameterized using a Visual Transformer (ViT). Each view within the panorama is fed into ViT individually as a sequence of tiles along with a learnable token called class (or CLS) following a known procedure. To obtain the viewpoint representation, the representation corresponding to the CLS token of each view is taken along with the mean across all views. The representation is discretized using a VQ-VAE bottleneck to obtain the final representation. A 6-layer Transformer with an embedding dimension of 256 is used. A Feedforward Network (FFN) is used after each attention operation of ViT. The FFN is a 2-layer Multilayer Perceptron (MLP) with Gaussian Error Linear Unit (GELU) activation, which first projects the input to a higher dimension D and then projects it back to the original dimension. We denote the dimension D of the FFN FFN Set to 512. Use 4 heads in ViT. Train the model for 20 epochs using Adam optimizer with a learning rate of 1e-4. The model is trained to predict the viewpoint as the next state of an action.

[0069] Figure 8 A graph of state simplicity scores for various machine learning (ML) models is shown by way of example. It can be seen that for the models shown, AC-State with and without an information bottleneck outperforms the other models shown in terms of state simplicity. AC-State with an information bottleneck has significantly higher state simplicity than AC-State without an information bottleneck.

[0070] Fig. 9 Plots of multiple accuracy metrics for various ML models are shown by way of example. It can be seen that for the models shown, AC-State with and without the information bottleneck performs at least as well as the other models shown in terms of accuracy of learned controllable latent states. It can be seen that for the models shown, AC-State with and without the information bottleneck outperforms the other models shown in terms of accuracy of ignoring extraneous noise. AC-State with the information bottleneck has significantly higher accuracy of identifying and ignoring extraneous noise than AC-State without the information bottleneck.

[0071] The controllable latent state accuracy is the viewpoint prediction accuracy of the current state. Exogenous Noise-Ignoring Accuracy is calculated as 1-(ε-33.33) / 66.67, where ε is the elevation prediction accuracy. Therefore, a higher elevation prediction accuracy leads to a lower exogenous noise induced accuracy. The proposed AC-State model has the highest controllable latent state and exogenous noise ignoring accuracy. Therefore, it is superior to the baseline considered when capturing controllable latent state information while ignoring exogenous noise. State simplicity is calculated as the number of ground truth states / number of discovered states. Therefore, lower state simplicity indicates a large number of discovered states, which means that the model cannot ignore exogenous information. The proposed model has the highest state simplicity, which demonstrates the effectiveness of the model in ignoring exogenous noise and capturing only controllable latent states.

[0072] Experiments show that AC-State reliably discovers controllable latent states across multiple domains. The dramatic simplification of controllable latent states discovered by AC-State enables visualization, precise planning, and rapid exploration. The field of self-supervised reinforcement learning has particularly benefited from these methods, and using AC-State is useful in a wide range of applications involving interactive agents as a self-contained module to improve sample efficiency given any task specification. As the abundance of sensors and the ubiquity of computing technologies (such as virtual reality, the Internet of Things, and self-driving cars) continue to grow, the ability to discover latent states that agents can control enables new classes of applications.

[0073] Asymptotic analysis of AC-State shows that AC-State recovers f*, the controllable latent state representation. The mathematical model considered is a deterministic exogenous block Markov decision process (Ex-BMDP). Using Ex-BMDP, the transition model of the latent state is decomposed into a controllable latent state that evolves deterministically, and a noise term (uncontrollable partial state). The noise term can be an arbitrary time-dependent stochastic process. If the reward does not depend on the noise, then any optimal policy can be expressed by the controllable latent state. In this sense, the recovered controllable latent state is sufficient to achieve optimal behavior.

[0074] Intuitively, Ex-BMDP is similar to a video game, where a "game engine" takes player actions and keeps track of an internal game state (a controllable state component), while visuals and sounds are rendered as a function of this compact game state. The core state of a modern video game is typically several orders of magnitude smaller than the entire game. AC-State recovery of an optimal controllable latent state involves (i) an action prediction term; and (ii) a mutual information minimization term. The action prediction term forces the learned representation to be captures information about the dynamics of the system. At the same time, the representation (optimal for action prediction) can also capture information that is not needed for control. In the analysis, it is assumed The unique coarsest solutions are those with discrete values ​​and exhibit controllable latent states.

[0075] To enable wider adoption in deep learning applications, the concept of roughness is generalized to minimizing the mutual information between x and f(x). These are related to data processing inequalities; a coarser representation reduces the mutual information with the input. Similarly, the concept of mutual information is general in that it does not require a discrete representation. Consider an exogenous block Markov decision process (Exo-BMDP), which is set up to model a system with exogenous and uncorrelated noise components. First, a block Markov decision process (BMDP) model is formalized. The BMDP consists of a finite set of observations With cardinality The potential state set With cardinality A finite set of actions Conversion functions Transformation function Reward Function And the starting state distribution The agent interacts with the environment and generates a single trajectory consisting of a sequence of latent states, observations, and actions (z1,x1,a1,z2,x2,a2,…,), where z1~μ(·), then x t ~q(·|z t ). The agent does not observe latent states (z1,z2,…,) but only receives observations (x1,x2,…,). If the supports of the emission distributions of any two latent states are disjoint, then the block assumption holds, For any potential state Finally, the proxy usage strategy Select an action.

[0076] AC-State can use a model called deterministic Ex-BMDP.

[0077] Definition 1 (Deterministic Ex-BMDP). A deterministic Ex-BMDP is a BMDP such that the latent state can be decoupled into two parts z = (s, e), where s ∈ S is a controllable state and e ∈ Ξ is an exogenous state. The transfer function is decoupled T(z′|z,a)=T(s′|s,a)T e (e′|e).

[0078] The above definition implies that there is a mapping from observations to corresponding controllable and exogenous potential states as well as Furthermore, the cardinality E of the exogenous latent states can be arbitrarily large. Note that the scene setting is not considered, and only access to a single trajectory is assumed.

[0079] Furthermore, it is assumed that the diameter of the controllable part of the state space is bounded.

[0080] Assumption 1 (Bounding diameter of the controllable state space). The length of the shortest path from any z1∈S to any z2∈S is bounded by D.

[0081] We now provide the structural results of the Ex-BMDP model. If it is not a function of exogenous noise, it is said to be an endogenous strategy. Formally, for any x1 and x2, if f * (x1) = f * (x2) Then π(·|x)=π((·|f * (x)).

[0082] Will Expressed as observing s′=f * (x) and following the policy π, a controllable latent state s′=f is observed at t time steps * The following results show that when the endogenous policy is executed, the distribution of the observation process on any x in the future t time steps has a decoupling property. This decoupling property proves that the controllable state partition is sufficient to represent the action prediction model.

[0083] Proposition 1 (Factorization property of endogenous policies). Assume x~μ(x) where μ is some distribution on the observation space and π is the endogenous policy. Then, for any t≥1, it holds

[0084] Observe that the assumptions used are twofold; namely, that π is an endogenous policy and that the initial distribution at time step t=0 is decoupled μ0(s,e)=μ0(s)μ0(e). In this case, the initial observations are conditioned on x. This also means that the latent state at the initial time step is deterministic, and therefore, the initial distribution is decoupled

[0085] The controllable partition is a Bayesian optimal solution. Consider a generative process where x is sampled from a distribution μ, and the agent executes a policy π for t time steps and samples x′. is expressed as a joint probability, and It is expressed as the probability that the action is a when observing x under this generative process. It is shown that the following result based on Proposition 1 shows the optimal Bayes solution equal for in is the probability for sample x.

[0086] Proposition 2. Assume that π is an endogenous policy. For some distribution μ, let x ∼ μ. Then, the Bayesian optimal predictor of the action prediction model is piecewise constant with respect to the controllable partition: for such that All t>0 and Established.

[0087] condition is necessary because otherwise the conditional probability Proposition 2 is proved via the factorization of the distribution of future observations into controllable and exogenous parts that holds when the executed policy does not depend on the exogenous state (Proposition 1).

[0088] Proof. The proof is performed by applying Bayes' theorem, Proposition (1), and eliminating terms from the numerator and denominator.

[0089] Fixed For any t>0, as well as The following relationship holds

[0090] Relation (a) holds by Bayes' theorem. Relation (b) holds by assuming that π is endogenous. Relation (c) holds by Proposition (1). Therefore, And the observation time is changed while the controllable latent state is fixed.

[0091] The coarsest partitioning is the controllable state partitioning. Proposition 2 shows that the partitioning caused by the controllable state The multi-step action prediction model is piecewise constant.

[0092] In this section, we assume that the policy executed is an endogenous policy that induces full exploration. By this assumption, we prove that there is no coarser partition of the observation space such that the set of inverse models is piecewise constant with respect to this partition.

[0093] The following assumptions were made regarding the strategy for collecting data.

[0094] Assumption 2. Let T D (s′|s) is obtained by executing the policy π D The Markov chain induced on the controllable state space, AC-State collects data through this strategy. 1. Markov Chain T D With fixed distribution μ D , so that for all s∈S and μ D (s,a)>0 and π D (a|s)≥π min . 2. Data Collection Strategy D From any state to all accessible states. For any s, s′∈S and any h>0, if s′ is reachable from s, then 3. Strategy π D It does not depend on exogenous state and is an endogenous strategy.

[0095] For further discussion of the class of Markov chains that rely on the stationary distribution assumption, see D.A. Levin, Y. Petres, Markov Chains and Mixing Time, vol. 107 (American Mathematical Society, 2017), chapter 1. The second and third clauses of Assumption 2 are satisfied for random policies that simply take random actions.

[0096] Consider a random process where observations are sampled from a distribution μ such that μ(s) = μ for all s D (s). The agent then executes policy π for t time steps D For simplicity, we denote the probability measure induced by this process as

[0097] Let's start by defining a few useful concepts. The set of controllable states that can be reached from state s in h time steps is represented as

[0098] Definition 2 (Reachable and controllable states). Let the set of controllable states reachable from state s∈S within h>0 time steps be R(s,h)={s′|max π P π (s′|s,h)=1}.

[0099] Due to the deterministic assumption of controllable dynamics, it is observed that every state reachable from s in h time steps satisfies amx π P π(s′|s0=s,h)=1.

[0100] Next, we define the notion of a consistent partition of a set of function values. Intuitively, if the function is piecewise constant on this partition, then the space The partition of is consistent with the set of function values.

[0101] Definition 3 (Consistent partitioning of G). Consider the set in Let f∶Y→[N] be expressed as A consistent partition is a partition where f(y′1)=f(y′2) for all y,y′1,y′2∈Y. (a,y,y′1)=g(a,y,y′2).

[0102] Observe that Proposition 2 shows According to f * Partitions and is consistent, because according to Proposition 2,

[0103] To verify that the coarsest abstraction according to the AC-State target is f * , consider the following definitions.

[0104] Definition 4 (Generalized inverse dynamic set AC(s,h)). Let s∈S, Let AC(s,h) be represented as the set of multi-step inverse models accessible from s in h time steps. Formally,

[0105] Observe that in Equation 4, because Inverse function is always well defined. and The inequality holds. This is established by assuming that: The stationary distribution at has positive support for all controllable states (Assumption 2). Inequality This is because, by definition, s′ is reachable from h′ time steps; Assumption 2 implies that policy π D Induces enough exploration so that

[0106] Theorem (f *is the coarsest partition consistent with the AC-State objective). Assume that Assumptions 1 and 2 hold. Then, for any s∈S, there is no ratio f that is consistent with AC(s,D) * Coarser partitioning.

[0107] Proof. Induction shows that for any h>0 and s∈S, for a set consistent with AC(s,h) In comparison, there is no A coarser partition. Since the set of reachable states in h = D time steps is S, and all states are reachable from any state in D time steps, this will directly imply that there is no ratio consistent with AC(s,D) Coarser partitioning.

[0108] Base case h = 1. Assume h = 1 and fix some s∈S. Since the controllable dynamics are deterministic, there is a state A reachable from s. Observe that for any Inverse Dynamic Satisfaction This can be demonstrated by applying Bayes' rule: The last relationship holds through Assumption 2. In addition, it is observed that due to That is, it is reachable from s, so the probability function is well defined.

[0109] Therefore, by Equation 5, for any There exists a∈A such that: Specifically, we choose a so that taking action a from s leads to s1′ and see that, by Equation 5,

[0110] Finally, by the fact that s∈S is an arbitrary state, the inductive base case is proved for all s∈S.

[0111] Inductive step. Assume that the inductive rule holds for all t∈[h], where h∈N. It holds for t=h+1. Fix some s∈S. The inductive step is proved to show is the coarsest partition of a consistent AC(s,h+1). This means that there exists Make and

[0112] Observe that by Definition 4, the following holds,

[0113] This means that the set AC(s, h+1) can be written as the union of the following sets: (1) and (2) for all nodes reachable from s in a single time step Collection The union of .

[0114] By inductive hypothesis, The coarsest partition that is consistent is It is only necessary to prove that Any There exists a∈A,h′∈[h] and And make This would mean that the set of reachable states in h+1 time states is also the coarsest partition consistent with AC(s,h+1).

[0115] Fixed of We show that there exists a certificate in AC(s, h+1) that distinguishes the two by considering three cases. 1. Case 1: and Both can be obtained from all In this case, for all All established By inductive hypothesis, we cannot combine and At the same time, it is consistent with AC(s′,h). 2. Case 2: Existence Make From s′, it is reachable in h time steps. is unreachable. Let a be the action that leads from state s to s′. In this case, according to the third assumption of Assumption 2, the following holds Relation (a) holds by Bayes' rule. Relation (b) holds by the fact that (s, a) deterministically leads to s'. Relations (c) and (d) hold by Assumption 2. Observe that Because it is assumed that it is not reachable when taking action a from state s Combining this fact with equation (7) means: Therefore, there is a distinction and The certificate between the two. Observe that because That is, it is reachable from s, holds, i.e., it is well defined. 3. Case 3: Existence Make From s′, it is reachable in h time steps. Unreachable. This case is symmetrical to case 2.

[0116] This confirms the result shown in Equation 6, and therefore, the induction and result hold. Reinforcement Learning with Rewards

[0117] The theory discusses the case where we want to learn a representation f(x) in the absence of rewards or other supervision. If the reward is only a function of the controllable latent state, then f(x) learned using the AC-State is sufficient for reinforcement learning using the reward signal. If the reward depends on both the controllable latent state and exogenous noise, but has an additive relationship, then the AC-State is sufficient to learn the optimal policy, but may lead to incorrect learning of the value function. In the case where the reward depends on both the controllable latent state and exogenous noise, using nonlinear interactions, the AC-State can be modified so that f(x) t ) predicts the reward. Intuitively, if one considers the problem of controlling a robot arm with a disturbing background TV, and considers the case where a reward is given when the robot turns left and the TV shows a specific frame. In this case, the controllable latent state needs to be made less coarse in order to capture the reward information at its latent state. At the same time, learning from the reward is not sufficient to learn the controllable latent state, because one can imagine a case where two controllable latent states are equivalent from the perspective of the value function, but are still different states.

[0118] Artificial intelligence (AI) is a field concerned with developing decision-making systems to perform cognitive tasks that traditionally require active actors (such as humans). Neural networks (NNs) are computational structures that loosely model biological neurons. Typically, NNs encode information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis of many AI applications, such as target recognition, device behavior modeling (as in this application), etc. Encoder 108, action predictor 112, or other components or operations of system 100, system 200, or method 300 may include or be implemented using one or more NNs.

[0119] Many NNs are represented as matrices of weights (sometimes called parameters) corresponding to the modeled connections. NNs operate by receiving data into a set of input neurons, which typically have many output connections to other neurons. At each traversal between neurons, the corresponding weights modify the input and are tested against a threshold at the destination neuron. If the weighted value exceeds the threshold, the value is weighted again, or transformed by a nonlinear function, the value is sent further down the NN graph, and if the threshold is not exceeded, the value is typically not transmitted to the next graph neuron, and the synaptic connection remains inactive. The process of weighting and testing continues until an output neuron is reached; the pattern and value of the output neuron that constitutes the result of the NN processing.

[0120] Most NNs rely on accurate weights for optimal operation. However, NN designers typically do not know which weights are appropriate for a given application. NN designers typically choose multiple layers of neurons or specific connections between layers that include circular connections. A training process can be used to determine appropriate weights by selecting initial weights.

[0121] In some examples, the initial weights may be randomly selected. The training data is fed into the NN and the results are compared to an objective function that provides an error indication. The error indication is a measure of how wrong the NN results are compared to the expected results. The error correction weights are then used. Over multiple iterations, the weights will converge together to encode the operational data into the NN. This process may be referred to as optimization of an objective function (e.g., a cost or loss function), thereby minimizing the cost or loss.

[0122] Gradient descent techniques are commonly used to perform objective function optimization. Gradients (e.g., partial derivatives) are calculated relative to layer parameters (e.g., aspects of weights) to provide the direction and possible degree of correction, but do not result in a single correction to set the weight to the "correct" value. That is, over several iterations, the weight will move toward the "correct" or operationally useful value. In some embodiments, the amount or step size of the move is fixed (e.g., the same from iteration to iteration). Small step sizes tend to take a long time to converge, while large step sizes may oscillate around the correct value or exhibit other undesirable behaviors. Variable step sizes can be tried to provide faster convergence without the disadvantages of large step sizes.

[0123] Backpropagation is a technique that feeds training data forward through a NN - here "forward" means that the data starts at the input neuron and follows the directed graph of neuron connections until it reaches the output neuron - and applies the objective function backward through the NN to correct the synaptic weights. At each step in the backpropagation process, the results of the previous step are used to correct the weights. Thus, the results of the output neuron correction are applied to the neurons connected to the output neuron, and so on, until the input neuron is reached. Backpropagation has become a popular technique for training various NNs. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0124] Fig.10 1 is a block diagram of an example of an environment including a system for neural network training. The system includes an artificial neural network (ANN) 1005 trained using a processing node 1010. The processing node 1010 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuit system. In an example, multiple processing nodes can be employed to train different layers of the ANN 1005 or even different nodes 1007 within a layer. Therefore, the set of processing nodes 1010 is arranged to perform the training of the ANN 1005.

[0125] The set of processing nodes 1010 is arranged to receive a training set 1015 of the ANN 1005. The ANN 1005 includes a set of nodes 1007 arranged in layers (shown as rows of nodes 1007) and inter-node set weights 1008 (e.g., parameters) between the nodes in the node set. In an example, the training set 1015 is a subset of the complete training set. Here, the subset can enable processing nodes with limited storage resources to participate in training the ANN 1005.

[0126] The training data may include a plurality of numerical values ​​representing a domain, such as image features, etc. Each value of the training or input 1017 classified after training the ANN 1005 is provided to a corresponding node 1007 in the first layer or input layer of the ANN 1005. The values ​​propagate through the layers and are changed by the objective function.

[0127] As noted, the processing node set is arranged to train the neural network to create a trained neural network. After the ANN is trained, the data input into the ANN will produce a valid classification 1020 (e.g., the input data 1017 will be assigned to a category). The training performed by the processing node set 1007 is iterative. In an example, each iteration of training the ANN 1005 is performed independently between the layers of the ANN 1005. Therefore, two different layers can be processed in parallel by different members in the processing node set. In an example, the different layers of the ANN 1005 are trained on different hardware. The members of the different members in the group of processing nodes can be located in different packages, shells, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between the nodes in the node set. This example is an additional parallelization, thereby independently training each node 1007 (e.g., neuron). In an example, the node is trained on different hardware.

[0128] Fig.11 A block diagram of an embodiment of a machine 1100 (e.g., a computer system) implementing one or more embodiments is shown by way of example. One or more of the encoder 108, hidden state focus operator 132, action predictor 112, loss operator 110, sensor 104, agent 102, method 300, or components or operations thereof may be implemented, at least in part, using components of machine 1100. An example machine 1100 (in the form of a computer) may include a processing unit 1102, a memory 1103, a removable storage device 1110, and a non-removable storage device 1112. Although an example computing device is shown and described as machine 1100, in different embodiments, the computing device may be in different forms. For example, the computing device may alternatively be a smart phone, a tablet, a smart watch, or include information about Fig.11 Other computing devices that may include the same or similar elements shown and described. Devices such as smart phones, tablets, and smart watches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of the machine 1100, the storage device may also or instead include a cloud-based storage device accessible via a network such as the Internet.

[0129] The memory 1103 may include volatile memory 1114 and non-volatile memory 1108. The machine 1100 may include or have access to a computing environment of various computer-readable media, such as volatile memory 1114 and non-volatile memory 1108, removable storage 1110, and non-removable storage 1112. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0130] The machine 1100 may include or have access to a computing environment, including input 1106, output 1104, and communication connection 1116. The output 1104 may include a display device, such as a touch screen, which may also be used as an input component. The input 1106 may include one or more of a touch screen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the machine 1100 or coupled via a wired or wireless data connection to the machine 1100, and other input components. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server, including cloud-based servers and storage devices. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), Bluetooth, or other networks.

[0131] Computer-readable instructions stored on a computer-readable storage device may be executed by a processing unit 1102 (sometimes referred to as a processing circuit system) of the machine 1100. Hard drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media (such as storage devices). For example, the computer program 1118 may be used to cause the processing unit 1102 to perform one or more methods or algorithms described herein. Additional Notes and Examples

[0132] Example 1 includes a method for controlling state learning, the method comprising: receiving a first observation in a time series of observations generated by a sensor and a second observation of the time series of observations, the second observation being separated from the first observation in the time series of observations by more than one other observation, the first observation being associated with a first action performed by an agent in response to the first observation; encoding the first observation and the second observation into corresponding first hidden state representations and second hidden state representations by a neural network (NN) encoder; combining the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation; operating the combined hidden state representation by an action predictor to obtain a second action; and changing the weights of the action predictor and the encoder based on the difference between the first action and the second action.

[0133] In Example 2, Example 1 also includes changing the hidden state representation to remove information not relevant to control of the agent, resulting in a focused hidden state representation, and wherein the action predictor operates on the focused hidden state representation.

[0134] In Example 3, Example 2 also includes, wherein changing the hidden state representation includes using a Gaussian variational information bottleneck, vector quantization, or a combination thereof.

[0135] In Example 4, Example 3 also includes, wherein changing the hidden state representation includes using the Gaussian variational information bottleneck and the vector quantization to generate the focused hidden state representation.

[0136] In Example 5, at least one of Examples 1 to 4 also includes, wherein changing the weight includes determining a probability using an objective function operating based on the first action, the first observation, and the second observation.

[0137] In Example 6, at least one of Examples 1 to 5 further includes receiving a value k indicating a difference between a first index of the first observation and a second index of the second observation in the time series of observations.

[0138] In Example 7, Example 6 also includes changing k to a different integer between consecutive training iterations.

[0139] In Example 8, Example 7 also includes, wherein k is an integer between one and a maximum path length in a Markov decision process that models the behavior of the agent.

[0140] In Example 9, at least one of Examples 1 to 8 also includes, wherein the first action and the second action are movements of the robot.

[0141] In Example 10, at least one of Examples 1 to 9 also includes, wherein the first action and the second action are operations of an agent utilizing a controllable latent state among a plurality of agents that do not have a controllable latent state.

[0142] In Example 11, at least one of Examples 1 to 10 further includes, wherein the first action and the second action are movement of the agent within the navigable environment.

[0143] Example 12 includes a system for controlling state learning, the system comprising a neural network (NN) encoder, the neural network (NN) encoder being configured to receive a first observation in a time series of observations generated by a sensor and a second observation in the time series of observations, the second observation being separated from the first observation in the time series of observations by more than one other observation, the first observation being associated with a first action performed by an agent in response to the first observation, and encoding the first observation and the second observation into corresponding first hidden state representations and second hidden state representations; a hidden state focus, the hidden state focus being configured to combine the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation; an action predictor, the action predictor being configured to operate on the combined hidden state representation to obtain a second action; and a loss operator, the loss operator being configured to change the weights of the action predictor and the encoder based on the difference between the first action and the second action.

[0144] In Example 13, Example 12 also includes, wherein the hidden state focus is further configured to change the hidden state representation to remove information irrelevant to control of the agent, resulting in a focused hidden state representation, and wherein the action predictor operates on the focused hidden state representation.

[0145] In Example 14, Example 13 further includes, wherein changing the hidden state representation includes using a Gaussian variational information bottleneck, vector quantization, or a combination thereof.

[0146] In Example 15, at least one of Examples 12 to 14 also includes a trainer configured to receive a value k indicating a difference between a first index of the first observation and a second index of the second observation in the time series observations, and between consecutive training iterations, change k to a different integer between a maximum path length and a maximum path length of a Markov decision process of a model behavior of the model.

[0147] Example 16 includes a (e.g., non-transitory) machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for controlling state learning, the operations comprising: receiving a first observation in a time series of observations generated by a sensor and a second observation in the time series of observations, the second observation being separated from the first observation by more than one other observation in the time series of observations, the first observation being associated with a first action performed by an agent in response to the first observation; encoding the first observation and the second observation into corresponding first hidden state representations and second hidden state representations by a neural network (NN) encoder; combining the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation; operating the combined hidden state representation by an action predictor to obtain a second action; and changing the weights of the action predictor and the encoder based on the difference between the first action and the second action.

[0148] In Example 17, Example 16 also includes, wherein the operation further includes: receiving a value k indicating the difference between a first index of the first observation and a second index of the second observation in the time series observations; changing k to a different integer between consecutive training iterations, wherein k is an integer between one and a maximum path length in a Markov decision process, wherein the Markov decision process models the behavior of the agent.

[0149] In Example 18, at least one of Examples 16 to 17 also includes, wherein the first action and the second action are movements of the robot.

[0150] In Example 19, at least one of Examples 16 to 18 further includes, wherein the first action and the second action are operations of an agent utilizing a controllable latent state among a plurality of agents that do not have a controllable latent state.

[0151] In Example 20, at least one of Examples 16 to 19 further includes, wherein the first action and the second action are movement of the agent within the navigable environment.

[0152] In some embodiments, the operations, functions or algorithms described herein can be implemented with software. Software can include computer executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transient memories (e.g., non-transient machine-readable media) or other types of local or networked storage devices based on hardware. In addition, such functions can correspond to subsystems, which can be software, hardware, firmware or a combination thereof. Multiple functions can be performed in one or more subsystems as needed, and the described embodiments are only examples. Software can be executed on a processing circuit system, for example, a digital signal processor, an ASIC, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA) or other types of processors operating on a computer system (such as a personal computer, a server or other computer system) can be included to turn such a computer system into a specially programmed machine. Additionally or alternatively, the processing circuit system may include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field programmable gate arrays (FPGAs), etc.). The terms computer-readable medium, machine-readable medium, and storage device do not include carrier waves or signals, as carrier waves and signals are considered too transient.

[0153] Although some embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the order or sequence shown to achieve the desired results. An ideal embodiment may include a user having confidence in the state of their data, settings, controls, and secrets before, during, and after migrating to a new version of an application. Using multiple factors to check data status, integrity, presence, and absence before and after migration can increase confidence. Other steps may be provided in the described processes, or steps may be eliminated, and other components may be added to or removed from the described systems. Other embodiments may be within the scope of the appended claims.

Claims

1. A method for controlling state learning, the method comprising: receiving a first observation in a time series of observations generated by a sensor and a second observation in the time series of observations, the second observation being separated from the first observation in the time series of observations by more than one other observation, the first observation being associated with a first action performed by an agent in response to the first observation; encoding the first observation and the second observation into respective first hidden state representations and second hidden state representations by a neural network (NN) encoder; combining the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation; operating, by an action predictor, on the combined hidden state representation to obtain a second action; as well as Weights of the motion predictor and the encoder are changed based on a difference between the first motion and the second motion.

2. The method according to claim 1, further comprising: The hidden state representation is altered to remove information not relevant to control of the agent, resulting in a focused hidden state representation, and wherein the action predictor operates on the focused hidden state representation.

3. The method of claim 2, wherein changing the hidden state representation comprises using a Gaussian variational information bottleneck, vector quantization, or a combination thereof.

4. The method of claim 3, wherein changing the hidden state representation comprises using the Gaussian variational information bottleneck and the vector quantization to generate the focused hidden state representation. 5 . The method of claim 1 , wherein changing the weight comprises determining a probability using an objective function operating based on the first action, the first observation, and the second observation.

6. The method according to claim 1, further comprising: A value k is received that indicates a difference between a first index of the first observation and a second index of the second observation in the time series of observations.

7. The method according to claim 6, further comprising: Between consecutive training iterations, k is changed to a different integer.

8. The method of claim 7, wherein k is an integer between one and a maximum path length in a Markov decision process that models the behavior of the agent.

9. The method of claim 1, wherein the first motion and the second motion are movements of a robot.

10. The method according to claim 1, wherein the first action and the second action are operations of an agent having a controllable potential state among a plurality of agents not having a controllable potential state.

11. The method of claim 1, wherein the first action and the second action are movement of an agent within a navigable environment.

12. A system for controlling state learning, the system comprising: The neural network (NN) encoder is configured as: receiving a first observation in a time series of observations generated by a sensor and a second observation in the time series of observations, the second observation being separated from the first observation in the time series of observations by more than one other observation, the first observation being associated with a first action performed by an agent in response to the first observation; as well as encoding the first observation and the second observation into respective first hidden state representations and second hidden state representations; a hidden state focus configured to combine the first hidden state representation and the second hidden state representation to obtain a combined hidden state representation; an action predictor configured to operate on the combined hidden state representation to obtain a second action; as well as A loss operator is configured to change weights of the action predictor and the encoder based on a difference between the first action and the second action.

13. A system according to claim 12, wherein the hidden state focus is also configured to change the hidden state representation to remove information that is not relevant to control of the agent, resulting in a focused hidden state representation, and wherein the action predictor operates on the focused hidden state representation.

14. The system of claim 13, wherein changing the hidden state representation comprises using a Gaussian variational information bottleneck, vector quantization, or a combination thereof.

15. A machine-readable medium comprising instructions which, when executed by a machine, cause the machine to perform operations for control state learning, the operations comprising the method of one of claims 1 to 11.

Images