A computing engine for autonomous, world-building, lifelong learning agents and general intelligence

Abstract

A system for determining domain observations of an environment is described. The system may include a domain engine module, an active sensing module, a fractal network module, and an executive agent module. The modules may be configured to execute a method for determining domain observations of an environment. The method may include generating or receiving a domain observation, generating or receiving a sim action, generating a fractal network associated with the domain observation or the sim action, generating an observation sequence from the fractal network, and comparing the observation sequence to the domain observation.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 362,466, filed April 5, 2022, which is incorporated by reference in its entirety. [Background technology]

[0002] The Cortical Computing Engine (CCE) is a new connectionist artificial general intelligence (AGI) framework. CCE improves on artificial neural networks (ANNs) by compensating for their inadequacies compared to known properties of biological brains and intelligence. CCE improves on desirable properties of machine intelligence such as common-sense reasoning, causality, generalization, and data efficiency, and by so doing, succeeds in avoiding critical deficiencies of ANNs. CCE is characterized by the following: (i) CCE grows parametric models, as opposed to assuming a fixed number of parameters; (ii) CCE employs local Hebbian-type learning, and thus can handle non-discriminatory models and software frameworks; (iii) CCE learns a continuous-time model of the world, as opposed to learning models batched into discrete sequences, i.e., learning a continuum for each time series; and (iv) CCE is a unified, naturally parallelizable cellular automaton-type algorithm, as opposed to a collection of heterogeneous predefined modules. CCE has already demonstrated significant cognitive capabilities for AGI and maintains desirable properties such as processing multi-modal data inputs simultaneously with spatiotemporal representations at diverse scales, generality and flexibility to adapt and learn over time, autonomy (including unspecified domains and unsupervised problems), interpretability enabling both inspection and introspection, and performance in a myriad of goal-oriented and explore-exploit settings. Summary of the Invention

[0003] In one aspect, a system for determining a domain observation of an environment is disclosed herein.

[0004] In some embodiments, the system may include a domain engine module configured to generate a first set of domain observations from the environment.

[0005] In some embodiments, the system may include an active sensing module configured to (i) receive a first set of domain observations, (ii) generate a second set of domain observations from the first set of domain observations, and (iii) generate a first set of sim actions.

[0006] In some embodiments, the system may include a fractal network module configured to (i) receive a second set of domain observations, (ii) receive a first set of SIM actions, (iii) generate one or more fractal networks associated with the second set of domain observations and / or the first set of SIM actions, (iv) generate an observation sequence from the one or more fractal networks, and (v) compare the observation sequence with the first set of domain observations.

[0007] In some embodiments, the system may include an execution agent module configured to (i) generate a second set of sim actions based on the observation sequence, and (ii) send the second set of sim actions to the active sensing module.

[0008] In some embodiments, the second set of sim actions causes the active sensing module and / or the domain engine to generate another set of domain observations and / or sim actions.

[0009] In some embodiments, the domain engine module generates a first set of domain observations at one or more rates or at one or more events.

[0010] In some embodiments, the first set of domain observations includes one or more temporal observations, spatial observations, or spatiotemporal observations.

[0011] In some embodiments, the first set of domain observations includes one or more visual observations, textual observations, auditory observations, vector observations, or scalar observations.

[0012] In some embodiments, the second set of domain observations comprises a transformation of the first set of domain observations.

[0013] In some embodiments, the active sensing module includes one or more active sensing modules, each active sensing module including a focusing head module configured to focus on or select a first portion of each domain observation, a data architecture module configured to generate nodes associated with the first portion of each domain observation, and a perception module configured to generate perceptual observations associated with a second portion of each domain observation, the second portion being proximate to the first portion of each domain observation.

[0014] In some embodiments, the number of one or more active sensing modules is dynamically changed in real-time based on receiving at least new types of domain observations.

[0015] In some embodiments, the fractal network module includes one or more fractal networks, each of which comprises a compute unit configured to store one or more placeholder values ​​associated with the operation of the respective fractal network.

[0016] In some embodiments, the number of one or more fractal networks changes dynamically in real time based on receiving at least new types of domain observations.

[0017] In some embodiments, each of the one or more fractal networks is grown independently of another fractal network to optimize capture of the observation sequence.

[0018] In some embodiments, each of the one or more fractal networks is iteratively trained using a first set of domain observations, a second set of domain observations, a first set of sim actions, or a second set of sim actions.

[0019] In some embodiments, each of the one or more fractal networks is iteratively trained to reach a confidence level associated with determining a relationship between each domain observation in the first set of domain observations and a sequence of observations.

[0020] In some embodiments, each of the one or more fractal networks is configured as an elastic architecture having one or more regions arranged in a linear or non-linear manner.

[0021] In some embodiments, each of the one or more fractal networks is configured to change a position of one or more regions in the elastic architecture to another position of the one or more regions in the elastic architecture.

[0022] In some embodiments, the computational units of one region are dynamically connected to the computational units of other regions during operation of the fractal network module using one or more horizontal electrical connections or one or more vertical electrical connections.

[0023] In some embodiments, each computational unit of one or more regions is comprised of one or more placeholders associated with arithmetic or logic operations.

[0024] In some embodiments, the execution agent includes a synchronous non-focusing agent.

[0025] In some embodiments, the execution agent includes a synchronous focusing agent.

[0026] In some embodiments, the execution agent includes an asynchronous focusing agent.

[0027] In some embodiments, the executive agent utilizes counterfactual control to instruct the fractal network module to determine future sim actions or future observations.

[0028] In some embodiments, the environment includes a computational domain or a cyber-physical domain.

[0029] In some embodiments, the transformation comprises a reduction in resolution or an amplification in resolution of the first set of domain observations.

[0030] In some embodiments, the second set of domain observations occurred simultaneously with the first set of domain observations.

[0031] In some embodiments, the data architecture module comprises a quad-tree structure, an oct-tree structure, a kd-tree structure, or any combination thereof.

[0032] Further aspects and advantages of the present disclosure will become readily apparent from the following detailed description, in which only exemplary embodiments of the present disclosure are shown and described. As will be understood, the present disclosure is capable of other and different embodiments, and its various details can be modified in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

[0033] Incorporation by Reference All publications, patents, and patent applications mentioned herein are incorporated by reference herein to the same extent as if each individual publication, patent, or patent application was specifically and individually incorporated herein by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the present disclosure contained herein, the present specification is intended to supersede and / or take precedence over any such conflicting material. [Brief description of the drawings]

[0034] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings. [Figure 1] 1 illustrates a non-limiting example of a Cortical Computing Engine (CCE) system as described herein. [Diagram 2] 1 illustrates a non-limiting example of a fractal network module as described herein. [Diagram 3] 1 illustrates a non-limiting example of an f-net computational unit as described herein. [Figure 4] unit

[0035]

number

[0036]

number

[0037]

number

[0038] While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, changes, or substitutions may be made without departing from this disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein may be employed.

[0039] A system for determining domain observations of an environment is described. The system may include a domain engine module, an active sensing module, a fractal network module, and an executive agent module. The modules may be configured to execute a method for determining domain observations of an environment. The method may include generating or receiving domain observations, generating or receiving sim actions, generating a fractal network associated with the domain observations or sim actions, generating an observation sequence from the fractal network, and comparing the observation sequence to the domain observations. In some cases, the environment includes a computational domain or a cyber-physical domain.

[0040] 1.0 Motivation Interest in artificial neural networks (ANNs) has risen again in recent years (2010s) after an initial surge (mid-1970s) and subsequent decline. As a subclass of gradient-based supervised machine learning methods, ANNs have been known for decades to be trainable on examples using the application of the differentiation chain rule to fit labeled examples to a given parametric model. Recently, ANNs have become attractive as they have benefited greatly from the abundance of labeled training data, as well as the development of semi-dedicated and fully-dedicated hardware to accelerate the aforementioned differentiation process. And the replacement of the standard sigmoid activation function with a rectified half-linear unit allows easier gradient propagation through the network, making it possible to train deeper ANN architectures end-to-end. These fundamental advances in training ANNs have then led to more efficient training of approximate dynamic programming reinforcement learning agents that use ANNs to approximate their underlying state spaces. Of the modern methods that use this principle, policy-gradient methods such as Asynchronous Advantageous Actor-Critic (A3C) or value-gradient methods such as Deep Q-Learning(r) (DQN) have gained the most traction and have demonstrated their soundness in a wide range of domains, from Atari(r) 2600 games to the ancient game of Go.

[0041] Despite the remarkable success in applying ANN-equipped Approximate Dynamic Programming (ADP) algorithms to various video games, their applicability to solving arbitrary problems out of the box required by AGI may be questioned. For example, even the individual visual processing CNN modules that often accompany ANN-based agents can be easily fooled by changes in the hue or saturation of the scene, rotation, or even the modification of a single pixel. And while this is obviously unfortunate, it may also be dangerous and detrimental to the rate of progress the scientific community is making in pursuit of AGI. As more computational and human resources are pooled and devoted to just one approach (to make ever-incremental progress in the pursuit of AGI), alternative approaches will obviously suffer the deprivation of these resources.

[0042] To better illustrate the risks and motivate the urgent need for resource allocation to benefit alternative approaches to AGI, in the following at least ten limitations of ANN-based ADP algorithms are enumerated that are believed to make ANN-based approaches towards AGI rather impractical.

[0043] 1. Challenge: Task Ignorant Model Architectures. The analysis of ANN limitations begins with a focus on the very first thing to keep in mind when talking about ANNs: network architectures, with regard to their applicability to AGI. Recently, a lot of effort has been put into solving the AGI problem and proposing fancy architectures (isolated AGI sub-problems, e.g., vision, control, etc.) that excel at a given benchmark problem, with a general agreement that there is no "one-shoe fits all" design when it comes to designing architectures. In fact, there has been an open dilemma about whether to adopt generic architectures (e.g., fully connected ResNets, where the trained network itself can choose the optimal connections and subset of layers), which unfortunately have slower convergence, or problem-specific architectures (e.g., ConvNets and RNNs, which employ reusable receptive fields over a selected number of layers, which are optimal for a given problem) that are much easier to learn. Unfortunately, future AGI agents will likely tackle new tasks not seen before (e.g., tasks that are quite different from those in the small set of tasks the agent saw during training), and the agent architecture may not be optimal for those new tasks. That is, even if the agent were allowed to retrain itself (adjust its ANN weights) during evaluation, its existing problem-agnostic architecture would almost certainly not be best suited to the new challenge at hand.

[0044] In essence, the ANN model architecture (as well as the initial values ​​of its parameters) may be chosen and fixed before ever seeing any training data, so the network architecture will rarely be optimal for the given problem at hand. This is the opposite of our proposed approach, where we start with a blank FPGA-type computational fabric and let the algorithm actually grow a network of relevant connections to optimally encode the data presented to it. Surprisingly, the brain cortex appears to employ a similar learning principle: the number of cortical connections in newborns is not fixed, but instead grows rapidly until the infant is about three years old.

[0045] 1.2 Challenges Task Interference through Model Inflexibility. The hope of researchers who choose to pursue an ANN-based roadmap to GI is that an agent trained on a sufficiently large number of diverse tasks will be able to generalize to essentially cover all remaining tasks the agent may encounter. However, if it turns out that the agent encounters a new task that it struggles to solve, or if it is desirable to adapt to changes in the non-stationary domain, the agent can be retrained. That is, the ANN controlling the agent is presented with new training data and its learning rate is increased accordingly. And while this is in fact what allows the agent to learn how to handle new tasks, the agent runs the risk of forgetting how to handle old tasks, since the optimized ANN used to encode how to handle old tasks must share its memory with the encoding of how to handle new tasks. In essence, the new agent knowledge encoded in the ANN will interfere with the old knowledge and affect the agent's overall performance. For example, an agent may be first trained to acquire the skill of driving manual cars in the United States (where the stick is operated with the right hand) and later (re)trained to acquire the skill of driving manual cars in the United Kingdom (where the stick is operated with the left hand). These two learned skills may begin to significantly interfere with each other, which may result in a non-zero probability that the agent will want to use both hands simultaneously to manipulate the stick.

[0046] As an alternative strategy for the agent to handle new tasks without forgetting how it handled the old tasks, the agent could be commanded to freeze its old ANN and couple it with an ANN built for the new task at hand. (Note that such coupling would require adding not only the parameters of the new ANN, but potentially also inter-ANN parameters from the neurons of the old ANN to the neurons of the new ANN, in the hope of reusing the already optimized filters of the old ANN during training of the new ANN.) However, given the exponential growth in the number of ANNs of parameters that comes with each new task to be learned, and the vast number of novel real-world problems that an AGI agent could potentially encounter, such an expansionist strategy is unlikely to be scalable. (This strategy would also be inconsistent with learning at the meta-level, since the AGI agent itself should be able to discover whether and when to expand its network to handle new tasks adequately.)

[0047] These are two important limitations of ANN-based AGI agents: Interference or inefficient learning of new tasks after agents are deployed is a direct consequence of the flexibility of ANN models; that is, being fixed-size monolithic black-boxes, ANNs cannot be trivially expanded to handle new tasks in a stable and scalable manner. Indeed, multiple ANNs can be composed together to form larger ANNs, but they cannot be expanded in a lightweight way to handle only the unique features of the new task at hand. (A good analogy here is with object-oriented programming languages, where new classes can be created using both composition of existing classes and lightweight extension / inheritance from existing classes. The latter approach is particularly efficient, as it results in new subclasses that share common characteristics of a superclass that apply its distinguishing set of features or patches.)

[0048] The assumption here is that a modern connectionist approach to AGI should assume an elastic rather than inflexible network architecture, and thus have the ability to slightly expand it when desired (e.g., increment the number of network layers or allocate computational units / neurons to it) and contract it when possible (free up infrequently used computational units / neurons for later reuse). In a sense, the computational fabric (e.g., substrate) of a modern AGI approach should act similarly to an FPGA or a biological cerebral cortex, with the corresponding programmable logic blocks or cortical micro-columns having the ability to be assigned and reallocated to continuously expanding and contracting models, respectively. An AGI system designed in this way should continually reuse unused portions of the computational fabric and use them to create patches of existing models to accommodate new tasks or changes in old tasks to which the existing models were not adequately addressed.

[0049] 1.3 Challenges: Unidirectional Pass Through the Model Further compromising the scalability of ANNs is the fact that the primary information flow through an ANN is the polar opposite of the information flow through the cerebral cortex: whereas in an ANN, information first enters the input layer and then flows unidirectionally through the network (possibly in repeated cycles) until it reaches the output layer at the other end of the network, information in the brain cortex flows bidirectionally, starting and ending at the very same input / output layer at one and only one end of the network (the other end is unspecified / open).

[0050] Specifically, in the cerebral cortex, raw sensory information enters the lowest layer of the cortical hierarchy of layers, is processed and integrated into the computational units below (groupings of neurons called mini-columns), and, if desired, is sent to higher layers for further processing. This operation is then repeated at higher layers in the hierarchy until the information reaches some desired higher layer, for example in the prefrontal cortex, where abstract information integration and higher level planning takes place. The information flow responsible for executing the planning then descends the cortical hierarchy of layers, becoming less and less abstract, where the information leaving the cortex is a stream of low-level motor commands that trigger corresponding actuators.

[0051] There are two immediate implications of such an open-ended bidirectional information flow strategy employed by the cerebral cortex that are crucial for constantly learning AGI agents. First, because information flow does not always have to pass through all pre-specified layers in the hierarchy (unlike ANNs), but only goes up to a desired task-specific level and then down from there, a continuously learning agent does not need to worry about its network being too shallow or too deep for the various tasks it will encounter. Second, the agent can always stack extra layers on top of its existing network to potentially boost the agent's performance, or even remove some layers from the top of the network to reduce the network memory footprint while only gradually impairing the agent's performance.

[0052] To build a continuously learning AGI agent, the primary information flow strategy in the new algorithm for AGI should follow the information flow strategy described above in the cerebral cortex. It should be bidirectional as well, and should start and end at the same network I / O layer at the bottom of the hierarchy of layers. Thus, the agent can similarly expand or contract its network to maximize the agent's performance on various tasks simultaneously while maintaining a desired memory footprint.

[0053] 1.4 Challenges: Bounded Temporal Credit Assignment Window Of particular importance for an AGI agent is to be able to make decisions in the context of its observations from potentially any past. This is particularly problematic for ANN-based agents, since it has been known for many years that ANNs currently suffer from the temporal credit assignment problem, which is not only relevant for feedforward ANNs, but also, unfortunately, for recurrent ANNs with memory cells that attempt to overcome it.

[0054] It is easy to understand why a feedforward ANN may not store observations from any past. The network output is conditioned only on the network input, which is itself of only limited size, as defined by a given network architecture. As a result, only a fixed number of observations can enter it into the network input layer, which results in the underlying time window having either a fixed time span or ad-hoc temporal gaps. And while the size of this time window can, in theory, grow exponentially with the network depth (as demonstrated in the Google® WaveNet ANN architecture), in practice, the amount of memory that such an ANN architecture defines to encode a particular time dependency is likely to be fixed and highly insufficient for any given time credit allocation problem at hand.

[0055] The reason why recurrent ANNs cannot remember relevant information from the past is more subtle. Recall that recurrent ANNs are similarly trainable using the application of the differential chain rule, and therefore also require a given fixed-size input vector. What is happening is that the recurrent ANN during training forms a feed-forward network of T modules with shared weights that can be evolved through T time steps and given T successive chunks of input vector data, corresponding to T successive time steps. What the network then learns is essentially how to best optimize its parameter space given independent training examples, each spanning no more than T successive time steps. The consequence of such a learning strategy is that if there is any temporal correlation between two observations separated by more than T time steps, these observations cannot be together part of any training example. Thus, there can be no reason (or opportunity) for the optimization process to encode said correlation in the parameter space of the trained model. (This problem can sometimes be alleviated by initializing the RNN neurons with the results of that preprocessing of some more initial data points of the underlying time series, but this strategy can in practice only work for very short time dependencies due to the problem of the gradient vanishing over time.) For example, if an ANN is trained to predict the external temperature in London at hourly intervals and T=24, the model will potentially learn that there is a day-night temperature cycle, but will not have the opportunity to learn that temperatures generally tend to be higher in summer relative to winter.

[0056] A direct consequence of the inability of ANNs to learn to always remember past relevant information is that ANN-based AGI agents such as A3C / DQN are often (relatively) memoryless. Their notion of what is relevant is premised on what was relevant during a narrow time window during agent training. As a result, agents may fail to condition the expected utility of their actions on all past relevant observations. To see this consequence, consider for example an A3C / DQN agent playing the game "Montezuma's Revenge®". Herein, the agent often finds itself in a room where the optimal policy is either to go one way (if the agent does not have the key) or to go the other way (if the agent has the key). Nevertheless, an agent whose policy is conditioned only on a few recent observations (downsampled to a resolution that cannot verify whether the agent has the key or not) has no choice but to follow a policy that does not know that the agent possesses the key. Similarly, a recurrent ANN-based agent is first trained on a toy-level curriculum where a short-term task of "collect the key then open the door" improves the chances that the network will register an event when the agent collects the key.

[0057] It is speculated that a brain approach to improving the temporal credit assignment problem need not rely on backpropagation, but instead may consist in circumventing the fixed time window problem by using a temporally delayed version of a local Hebbian-learning rule. 1.5 Challenges: Bottom-up Learning of Final Representations.

[0058] It is somewhat surprising that one of the most notable features of ANNs - their condensed representations of domain signals - is in fact neither biologically accurate nor attractive for AGI agents: the representations they learn are final (as opposed to refine-able over the course of the agent's life), difficult to transfer to new domains (because they are anchored to specific fine-grained sensory signals from the existing domain), and not robust (i.e. easily fooled), so a completely different approach is needed.

[0059] To explain the biological inaccuracy of current ANN representation learning, it is worth noting that biological brains do not simply perceive fine-grained signals all the time, and therefore may not employ bottom-up strategies such as first learning low-level fine-grained representations and then using them to learn higher-level representations. That is, raw signals (visual, auditory, somatosensory) first pass through dedicated signal aggregators and amplifiers (e.g., eye fixations on parts of the visible light spectrum or amplification of frequency bands on the cochlea), and thus most of the information the brain receives from body sensors is, in fact, very coarse resolution. And as long as this coarse resolution information is sufficient for the agent to achieve its purpose, no further signal amplification / refinement is required. For example, a spot on a car windshield is always visible, but most of the time it is not perceived by a trained driver who rarely chooses to amplify visual signals at such a short focal distance.

[0060] Only when signals perceived at a coarse resolution are no longer sufficient to achieve a given agent's objectives will the agent make an effort to perceive signals at a finer resolution. To this end, the agent will issue respective commands to its sensory signal amplification mechanisms to explore and expand selected parts of the sensory spectrum in the hope of encountering snippets of high-resolution signals that will provide useful information for the given task at hand. In other words, the agent will progressively build its internal representation of the environment in a top-down manner (as opposed to ANNs that build their representations in a bottom-up manner anchored at the bottom to high-resolution raw sensory signals), starting from approximations of blurry low-resolution signals and gradually elaborating them in greater detail, if necessary. As a result, domain objects that appear to be of uniform resolution will in fact be represented internally in the agent's brain with variable resolution, depending on the required sensory complexity of the task in which they were involved.

[0061] In direct contrast to ANN representation learning strategies, the above brain strategy overcomes the problems of elaborability, transferability, and availability that plague ANN learning representations. First of all, note how ANN learning representations are non-elaborable. That is, only once ANN learning is finished, the parameters of the network are final (optimized for a given task and a given resolution at which the task feeds the ANN with input signals). As a result, if this resolution at which the task signals are input to the network is subsequently changed, the network will likely no longer function as intended. For example, an image classification ANN will quickly stop working if the resolution at which images are presented to it is doubled. (This is the opposite of, for example, the family of pseudo-Hilbert curves, which allow for gradual elaboration of signal representations and are not derailed as the signal resolution increases.) And it is precisely because of this non-elaborability of ANN representations that the network can possibly be retrained, which, unfortunately, may no longer be possible if AGI agents are already deployed in the field.

[0062] Transfer of learned representations, which is equally important for AGI agents, has also been a problem for ANN-based agents. Part of this is that high-resolution object representations, relative to coarse-grained approximations, are much harder to transfer between domains. For example, in an Atari 2600 learning environment, it would be difficult to transfer high-resolution car sprites from the Enduro game to the Pacman game, where the car sprites are gone and replaced with high-resolution sprites for creatures. However, if these two separate objects are represented using a resolution representation that can be refined (as is speculated to be the case in the cortex), then in fact transfer can be successful. For example, an agent that learns to avoid colliding with cars in Enduro, which are represented at low resolution as moving white blobs, can perform reasonably well when attempting to avoid being eaten by creatures in Pacman, which are also represented at low resolution as moving white blobs.

[0063] Finally, the fact that the ANN learning representation includes fine-grained signal filters at the lowest layer of the network exposes the network to adversarial attacks that are often imperceptible to a human observer. A cleverly designed attack exploits the amplification of errors in the activation of ANN filters by information propagating through the network. This involves a fine-grained, barely noticeable (including as little as one pixel), targeted perturbation of the input signal, which causes the lowest-layer network filter to misclassify the input, which in turn fools the higher-layer filters, and so on. This often results in surprising errors, such that a pair of images that appear identical to a human observer (we must remember that human observers perceive these images at low resolution, unless they choose to enlarge some parts of it) are recognized, with confidence approaching 100%, as belonging to completely different categories. In contrast, the human visual system would not be easily fooled by imperceptible input signal perturbations, since it first perceives the entire scene at low resolution and only later (if at all) chooses to recursively enlarge parts of the scene.

[0064] In summary, a viable representation learning system for AGI agents needs to generate representations that can be continually refined with improving signal resolution, are easily transferable to new domains, and are resistant to adversarial attacks. And while bottom-up processes for learning final ANN representations have led to many breakthroughs in machine learning, they have failed to generate representations that meet these three key requirements for AGI agents. What is needed to improve this situation is an approach for learning representations that is, at least in principle, similar to how we accomplish the task.

[0065] 1.6 Challenges: Episodic learning. Of the existing learning paradigms that encompass ANNs that are particularly impractical for either biological or AGI agents, episodic learning deserves special attention. Episodic learning, proposed to facilitate agent learning by decomposing a given agent domain into smaller self-contained domains, introduces two important limitations for future AGI agents.

[0066] The first of these limitations is easy to understand. The partitioning itself is inevitably ad hoc, since it is ultimately up to human task designers to decide how to distill the agent domain into smaller chunks, called episodes. There is therefore a risk that isolated episodes may not contain all the relevant information an agent would like to have so that it can learn all the skills it needs to succeed in the larger domain. For example, if an agent encounters a seemingly useless type A object (e.g., carbon dioxide) in one episode, whereas in another episode it encounters another seemingly useless type B object (e.g., hydrogen), the agent has no opportunity to learn to combine these two objects to produce a potentially useful type C object (methane, in this example). In contrast, a continuously learning AGI agent may encounter both seemingly useless objects (A and B) throughout its life, and by experimenting with combining them, be more likely to reveal that they are in fact key building blocks for manufacturing a useful object of type C.

[0067] A second limitation of AGI agents with episodic learning is that the underlying learning process appears non-stationary. To understand why, it is important to first recall that the agent's world is intended to always be reset before the start of each episode. This (arguably unrealistic) experimental design choice is intentional, and is extremely useful for ANN-based agents, as it removes the need to remember what happened in past episodes. (Note that ANNs are known to perform poorly in long-term memory tasks outside the training domain, as explained earlier.) However, from the perspective of a continuously learning agent, if the state of the world (including, for example, the agents that have changed in said world) is silently reset between episodes, then the entire learning domain appears to be non-stationary and essentially unlearnable. For example, in the Starcraft® video game, a continuously learning agent will remember that it has already collected all the crystals from a given part of the world, and will not choose to send a collector to that part of the world again without being explicitly told that the episode has restarted.

[0068] In essence, the ad-hoc human distillation of the agent domain into much shorter, seemingly self-contained episodes, which often facilitates the training of ANN-based agents, appears to be an artifact of non-AGI research. Not only does episodic learning potentially deprive agents of skills whose learning requires exposure to a continuous set of episodes, it also introduces fake domain non-stationarity that continuous-learning AGI agents would struggle to model. Thus, research in building AGI agents should not use the concept of episodes, either at training or testing time.

[0069] 1.7 Challenges: Data-Driven Learning on How to Reason. A successful AGI agent must not only be able to recognize input signal patterns and act on them reactively using responses extracted from its already trained ANN, but also engage in proactive reasoning via potential future planning as well as retrospective analysis of adverse events from the past. And while there are likely dedicated processes in the brain responsible for such strategic planning or counterfactual reasoning, none of these processes are explicitly encoded in the parameters of the ANN.

[0070] Indeed, the only way an ANN can approximate such inference (perhaps only within the narrow scope of a given training domain) is to make this inference inferable from the data on which the ANN is trained. Thus, exotic ANN architectures have been proposed to facilitate such implicit inference, with a somewhat similar overarching theme. They provide the ANN with an external memory and a controller (e.g., a recurrent ANN), allowing the latter to tinker with the contents of the former to achieve a desired objective for a given training task. The end-to-end trained, distilled controller network can then be interpreted as engaging in some kind of inference on the contents of the external memory. This implicit, narrow-domain inference can then be hoped to later generalize to other domains, eventually covering the entire space of inference processes if only a sufficient number of training tasks are involved.

[0071] Although this implicit ANN inference strategy is sound, it is arguably not scalable for future AGI agents because it would need to be pre-trained on a potentially very large number of rare inference tasks in order for future AGI agents to know how to handle them. For example, an agent might infer unusual tasks such as how to detach pantyhose from bike spokes. Such tasks are unlikely to be encountered by the average human, let alone add sufficient scale and variety to the training set of an ANN agent.

[0072] A more efficient and scalable strategy (inferred to be adopted by the brain) is to equip the agent's brain with explicit, general-purpose reasoning mechanisms such as counterfactual replay and preplay, and to defer the encoding of the actual reasoning process until required by the task. In essence, given the task at hand, a scalable AGI reasoning system needs to use its temporary memory to execute and encode the results of its available "what-if" replay and preplay tests that are applied to said task. Only the relevant results of these tests are then later copied into the agent's permanent memory, thus extending the agent knowledge base with additional "recipes" that the agent can easily (without conscious reasoning) apply to the above tasks in the future.

[0073] 1.8 Challenges: Unrealistic Agent Objectives A common objective of ANN-based AGI agents is to maximize the sum of discounted expected rewards collected over the course of several episodes during agent training. Therefore, an agent following such a strategy is simply myopic, for the following reasons. In theory, the Q-layer of a trained A3C / DQN integrates all the rewards that the agent collects as it follows the currently learned policy, but in practice this is not the case. In infinite horizon planning problems, the value of each Q-layer neuron may need to be bounded (to allow the gradients backpropagated through the network to be bounded), and each subsequent reward that the agent is expected to collect is discounted by a fixed γ<1. (The discounting effect is exponentially augmented for subsequent actions to converge and generate a bounded, geometric set of rewards as a result). As a result, A3C / DQN agents underestimate rewards that they expect to encounter in the distant future by a fairly ad-hoc factor, often with disastrous effects on the agent's performance. For example, in the Montezuma's Revenge game, if an agent is lured with a small, positive, immediate reward to enter a trap room after a sufficiently long delay (causing the agent to incur a huge reward penalty), the agent is likely to re-enter the room at the next opportunity (beyond the small, positive, immediate reward) because the long delay resulted in such a significant discount of the huge penalty. In essence, ad-hoc discounting for later events, while mathematically convenient, is simply something that humans or AGI agents should use. (Humans do in practice discount rewards from situations that are unlikely to occur but do not automatically correspond to situations that occur later in time.)

[0074] The total reward that an agent expects to receive is yet another impractical objective for ANN-based AGI agents. For one thing, it is simply impractical for an agent to try to plan its actions taking into account all rewards it can obtain over the course of its entire life. Not only does it require reasoning about an exponential-in-planning-horizon number of future possible reward trajectories, but it also results in unbounded sums of rewards for infinite planning horizons. And while the latter problem can be somewhat mitigated by using discount factors (leading to agent myopia, as discussed) or by using an average-reward objective function (which in turn generates far-sighted agents), the former problem will still persist, especially for long planning trajectories of real-world planning problems.

[0075] Thus, the above objectives of current ANN-based AGI agents, despite their mathematical convenience, appear to be rather impractical and biologically inaccurate. Specifically, the average biological actor is primarily interested in aversively avoiding the most painful or life-threatening experiences while at the same time opportunistically pursuing the most hedonic experiences over the entire span of the agent's life. Also, it certainly does not employ additional reward aggregation. For example, for the average human, the reward of eating an apple on a given day does not simply accumulate the more apples one eats, but rather is a function of the intrinsic activation of taste pleasure receptors and the deactivation of hunger pain receptors. In essence, not knowing how many times a given reward will be encountered on a given trajectory naturally leads the biological agent to attach disproportionate importance to rare, but more extreme, rewards on said trajectory. That is, unlike a typical A3C / DQN agent, which adds all rewards it encounters, such that rewards from rare but often significant events are diluted by many minor rewards (e.g., incremental costs for the agent's movements), biological agents effectively avoid this undesirable dilution. Also, the perceived inability of biological agents to adequately distinguish between the utility of reward trajectories involving different numbers of activations of the same rewarding stimulus can be largely mitigated by making the reasonable assumption that in real biological systems, repeated activation of the same stimulus is likely to induce activation of other (stronger) stimuli. (For example, repeated activation of sugar taste receptors increases the likelihood of activating stomach pain receptors.) As a result, the agent will implicitly only prefer trajectories with more similar positive rewards (or avoid trajectories with similar negative rewards), because the agent model suggests that such repeated rewards are more likely to be followed by other (stronger) types of rewards.

[0076] 1.9 Challenges Back-propagation of Reward Signals The use of ANNs as function approximation components of reinforcement learning algorithms, such as in the value network of DQN or the critic network of A3C, introduces an additional set of problems. To understand these problems, recall that DQN / A3C still belong to the class of supervised learning algorithms that can be trained using backpropagation. That is, they still require a supervision signal, computed as the difference between the network's prediction of the discounted expected reward of the agent performing an action and a more accurate prediction of said value, computed using the observed immediate reward for performing said action. They are thus directly exposed to a set of new problems, among which the following two are most prominent:

[0077] First, in the absence of immediate rewards provided to the agent by the simulator, the agent does not learn anything; i.e., the teacher signal is zero, and thus the corresponding gradient is back-propagated through the layers of the agent network. For example, in a maze domain where the agent is only rewarded if it finds the exit point of the maze, barring the use of any intrinsic motivation auxiliary rewards, the agent will not encounter any rewards for a long period of time and therefore will not learn anything (it will not update the parameters of its ANN). And while this may be an acceptable problem for an agent that has the comfort of living indefinitely in a simulated training domain, this problem will certainly be an intolerable problem for an AGI agent operating in a real-world environment. What is essentially proposed here is that AGI agents need to employ other forms of learning (rather than back-propagation of expected reward signals) to efficiently learn to operate in a given sparse-reward domain. In other words, learning of the underlying world model by the agent should occur even in the absence of any specific reward signal, and the role of perceived reward is limited to modulating the learning process and biasing the statistics of the underlying learning process towards more meaningful events.

[0078] Second, including agent rewards in the gradients that are back-propagated through the network has a significant impact on the later transferability of agent knowledge to new domains. Specifically, gradients carrying partial reward signals will, at some point, inevitably start injecting filter parameters, where the reward comes from the current given training task. And while this is rightly geared towards optimizing the agent performance for that very task, it results in the network filter parameters becoming permanently tuned to the current training task. That is, some input patterns that are meaningless to the agent’s performance for the old task (potentially important for future tasks) end up being filtered out by the network. As a result, future attempts to transfer knowledge of the original network (e.g., the bottom-layer convolutional filters) and reuse it in a new network built for a new set of tasks may simply fail. For example, an agent trained to drive a car will likely optimize its parameter space to infer rewards for its actions based on what it perceives on and around the road, but not on what is in the sky. So if this agent is transferred to a new domain where it is asked to predict the likelihood of rain based on what it currently sees, it will likely filter out all relevant visual cues that are clearly visible in the sky (e.g., the color and shape of clouds).

[0079] We therefore speculate that a successful AGI agent will have the ability to learn even in sparse or zero-reward domains, and will refrain from encoding the rewards of the domains in the model in a way that would prevent the agent's knowledge from transferring to new domains in the future. We speculate that this can only be achieved using an approach that does not rely on backpropagation of reward signals. 1.10 Challenges: Shallow Deep Reinforcement Learning

[0080] A rather subtle misconception that characterizes the A3C / DQN deep reinforcement learning family of algorithms is that they actually only use shallow reinforcement learning, even though they sit on top of a deep signal processing ANN architecture. And this somewhat misleading terminology would be less of a problem if it weren't for the constraints that the shallow reinforcement learning (RL) component of A3C / DQN implies. Here we describe some of the ones that are of particular interest to AGI agents.

[0081] First, shallow RL agents explicitly construct joint plans: atomic, non-decomposable plans that may implicitly include multiple subtasks, but that prescribe carefully selected (joint) actions to be performed at each time step to directly contribute to the fulfillment of the plan. And because plans are not explicitly decomposable, a (model-based) shallow RL agent that aims to accomplish a given task often needs to plan ahead for a prohibitively large number of intermediate steps; that is, it may plan all the steps of auxiliary subtasks that are interspersed among the actions of the agent's primary task at hand. For example, an agent that schedules its work meetings during the day may also plan ahead for arguably unrelated subtasks, such as which restroom to use and which salad dressing to choose at lunch. A more efficient strategy would be to (i) first explicitly discover and simultaneously track multiple independent trajectories of executable events, and then (ii) multitask among them toward constructing relatively short individual plans for each separate trajectory (similar to CPU multitasking between concurrent processes). This is essentially what the cerebral cortex is suspected to do, and what is proposed here.

[0082] Another limitation of shallow RL agents is related to their inability to (i) automatically discover and form plans of various levels of abstraction (operating in a more manageable plan space and facilitating plan transfer to new domains), as well as (ii) seamlessly switch between these plan abstractions in order to maximize the agent’s performance in a given domain. That is, as already noted, certain plan abstractions are readily available to biological agents that already perceive variable-resolution world observations (thanks to the selective signal amplifiers mentioned above). And once the agent manages to distill its trajectory of coarse-grained observations (potentially of different modalities), they may indeed constitute a solid basis for abstract plan formation. These coarse-resolution abstract plans can be complemented (at the agent’s whim) with finer plans formed from higher-resolution agent observations, allowing the agent planning mechanism to effectively switch between (or even merge) plan trajectories of different levels of abstraction. For example, an agent playing the video game capture the flag may develop an abstract plan (from low-resolution observations) for how to navigate the underlying environment to find the flag or home base. The very same agent may look at the subtle, small features of the other player characters in the game and automatically discover a more specific (from high-resolution observations) concurrent plan for how to distinguish between teammates and players on the opposing team. Not only will these plans be relatively short and uncomplicated, but they will also be much easier to transfer to slightly different domains with either different map layouts with different opponent types.

[0083] However, such automatic discovery of plan decomposition and plan abstraction is not at the heart of shallow RL agents, and therefore the plans these agents form typically have short horizons, poor transfer across domains, and are difficult to interpret.

[0084] In conclusion, the AI ​​community has seen a resurgence of interest in artificial neural networks since the early 2010s, thanks in large part to the abundance of labeled data and computational power that makes ANNs practical to train. Since then, ANNs have been applied to a variety of tasks, ranging from speech and image recognition to playing complex games such as Go and StarCraft, and have made impressive progress, positioning ANNs as potential AGI algorithms. And while the fact that ANNs are very data-intensive has been known for some time, it is only in recent years that researchers have uncovered other shortcomings of ANNs amid the various ANN roadmaps to AGI. This section has highlighted the top ten of these shortcomings. The remainder of this document describes how to remedy them once and for all using the Cortical Computing Engine approach to AGI.

[0085] 2.0 Cortical Computing Engine 2.1 Background. The Cortical Computing Engine (CCE) is a refinement of Artificial Neural Networks, Capsule Networks, Hierarchical Temporal Memory, Piaget's Schemas and Constructivist Theory, and Generalized Value Functions.

[0086] 2.2 Glossary. The symbols used in the description of CCE are introduced appropriately in subsequent sections. For convenience, some or all of the symbols are described herein.

[0087] r i : f Net area i.

[0088] R:f Net top region identifier.

[0089] Rmax : Practical limit on allocatable f Net space.

[0090] L(r): The current number of allocated compute units in region r.

[0091] A(r): The index of the unit currently being attended to in region r.

[0092]

[0093]

number

[0094]

[0095]

number

[0096]

number

[0097]

number

[0098]

number

[0099]

[0100]

number

[0101]

[0102]

number

[0103]

number

[0104]

[0105]

number

[0106]

number

[0107]

[0108]

number

[0109]

number

[0110]

[0111]

number

[0112]

[0113]

number

[0114]

[0115]

number

[0116]

[0117]

number

[0118]

number

[0119]

number

[0120]

[0121]

number

[0122]

number

[0123]

number

[0124]

number

[0125]

number

[0126]

[0127]

number

[0128]

number

[0129]

[0130]

number

[0131] a: f Sim actions corresponding to some f Net units in r1.

[0132] o:r1 are f Sim observations corresponding to some f Net units.

[0133]

[0134]

number

[0135]

number

[0136]

[0137]

number

[0138] 2.3 System Overview. The Cortical Computing Engine (CCE) consists of (i) a Fractal Network (f Net) module, (ii) a Focusing Simulator (f Sim) module, and (iii) an Executive Agent (EA), as shown in FIG. 1. The high-level information flow in the CCE is as follows: The Domain Engine constantly generates domain observations (at whatever maximum temporal and spatial resolution the Domain Engine can generate) given the domain actions currently received at its input. The domain observations may generally be multimodal, and thus there is a separate Focusing Simulation module associated with each domain engine modality. For example, a domain engine that generates visual and auditory domain observations sends them to a visual-f Sim and an auditory-f Sim, respectively. Each f Sim may be further subdivided to handle independent aspects of a given modality, as described elsewhere herein.

[0139] The main role of the focusing simulator module is to selectively reduce the resolution of domain observations of a given modality. That is, each f Sim has a focusing head (controllable via f Sim actions) that amplifies the resolution of a selected subset of the spectrum of the observation modality while downsampling observations outside this subset. (This procedure can be applied recursively for smaller subsets of the spectrum subset, etc.). Such downsampled domain observations are then clustered into discrete f Sim observations and passed to the fractal network module along with their corresponding f Sim actions. For example, an auditory-f Sim focusing head can focus only on a narrow audio frequency band (and only within it perceive audio frequencies with high resolution) while simultaneously downsampling auditory frequencies from outside this band. Then, since the information required to describe the auditory signal in these narrow high-resolution and wide low-resolution bands is significantly reduced, the auditory f Sim can more efficiently discretize them into an acceptable format by the fractal network module. (It is worth noting that the set of fSim modules can be extended at CCE runtime if the domain engine starts providing new types of domain observations.)

[0140] Fractal network modules operate independently. They each receive the same sequence of discrete f Sim actions and observations and independently evolve their underlying networks to best capture said sequences. What distinguishes f Net modules is how quickly they learn the underlying sequences and how aggressively they attempt to compress the sequences. For example, a fast-f Net can be specified to learn the underlying sequences in a single exposure but provide very little sequence compression and very limited sequence generalization, whereas a slow-f Net can be specified to provide much better compression and generalization even after being exposed to the underlying sequences multiple times. Thus, a good CCE design strategy is to assume a diverse set of Net modules (although the composition of this set need not be fixed and can be changed as desired during CCE run-time). From incoming sequences of f Sim actions and observations, the f Net then continuously learns, refines, and updates its internal world model, which is used by executive agents to constantly predict the effectiveness and usefulness of f Sim actions.

[0141] The executive agent is responsible for adjusting the f Sim observation and f Sim action utilities predicted by each f Net independently and determining the f Sim action that best achieves a given objective (e.g., the total future domain reward generated by the domain engine). Alternatively, if at a given time none of the f Sim actions result in the desired objective change and no incoming f Sim observations are registered, the executive agent itself can instruct the f Net to imagine various plausible (or counterfactual) future f Sim actions and f Sim observations. This can then cause the CCE to execute preemptive action plans to best prepare the CCE for these imagined plausible future outcomes.

[0142] Finally, the f Sim actions sent by the executive agent to the f Sim module can either control the focusing head of some f Sim module or execute some domain actions directly. In the former case, a change in the position of the f Sim focusing head triggers a new f Sim observation to be sent to the f Net module, while in the latter case, the domain action is executed by the domain engine until a new f Sim observation is registered by some f Sim focusing head. Note the asynchronous execution of the domain engine and the CCE. Some domain observation changes may be imperceptible by the CCE f Sim module (given the current position of the f Sim focusing head) and therefore do not trigger any response from the CCE. Similarly, the executive agent action of moving the Sim focusing head is executed asynchronously with the domain engine process that generates the new domain observation.

[0143] 3.0 Fractal Networks This section presents the architecture of the fractal network modules and their core functionality that are invoked by the CCE execution agent when performing cognitive tasks.

[0144] 1 Architecture The description of the fractal network architecture begins with an overview of the fractal network module and then provides details regarding the micro-architecture of the computational unit on which the fractal network module is built.

[0145] 3.1.1 fNet module. In contrast to artificial neural networks (ANNs), where the network architecture is fixed, fractal network modules exhibit architectural elasticity. An exemplary fractal network module is shown in Figure 2. The f Net consists of a set of regions (aqua boxes): r1, r2, ..., r R are stacked on top of each other in a linear hierarchy, with r1 being the bottom region of the f Net and rR being the current top region of the network. Unlike ANNs, where the input and output regions are separate, in f Net, region r1 is responsible for both the input and output from the f Net modules. Furthermore, f Net has a flexible number of regions in that it simply starts with r1 and later adds new regions on top of existing regions or removes regions from the top if necessary. For example, the f Net in Figure 2 consists of regions r1, r2, r3, and the current top region r4.

[0146] The f Net region itself also exhibits elasticity. Each region r is assigned to the same computation unit

[0147]

number

[0148] The connectivity between Net computational units is similarly elastic: computational units in region r (white boxes with black outlines) can be connected to each other using directed horizontal links (black arrows), where each such horizontal connection can further have an associated vertical connection to a dedicated computational unit in region r+1 (black line with black circles at each end). That is, each unit in region r+1 can have exactly one vertical connection to some horizontal connection in region r directly below region r+1. In FIG. 2, some ordered pairs of units are connected by dedicated directional horizontal connections (e.g.,

[0149]

number

[0150]

number

[0151]

number

[0152]

number

[0153]

number

[0154]

number

[0155]

number

[0156]

number

[0157]

number

[0158]

number

[0159] The horizontal and vertical connections between the f Net computational units are not initially present in the f Net module. Only once CCE is performed do the f-nets begin to incrementally expand their horizontal and vertical connections (or prune them later, if desired, for memory optimization purposes). This is in stark contrast to the uniform microarchitecture within the underlying compute units, which remains fixed throughout the entire CCE run time, as described herein.

[0160] 3.1.2 f Net computational units. The microarchitecture of the f-Net computational unit is fixed and consists of internal switches and external connections, as shown in Figure 3. In contrast to ANN neurons, which are assigned to ANN layers at compile time, Net computational units are dynamically assigned to Net regions at runtime. Already assigned units

[0161]

number

[0162]

number

[0163]

number

[0164]

number

[0165]

number

[0166]

number

[0167]

number

[0168] In addition to this unique vertical connection, the computational unit

[0169]

number

[0170]

number

[0171]

number

[0172]

number

[0173]

number

[0174]

number

[0175]

number

[0176]

number

[0177]

number

[0178]

number

[0179]

number

[0180] As shown in Figure 3, The F Net computational unit is depicted with all its theoretically possible connections and switches. For the purposes of practical explanation, in the following it is assumed that if an f Net horizontal connection has a weight of 0, it is simply not depicted in FIG. In addition, if a horizontal connection has a positive weight but is still not strong, it is drawn using a dotted arrow, and the number above this arrow is the weight of the connection. Similarly, if vertical connections have not yet been formed, they are not depicted in FIG. Furthermore, in the presence of multiple sequences represented internally in an fNet, different colors are used, where possible, to represent the internal connections of the different sequences. Finally, the unit is a complete descriptor.

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185] In stark contrast to ANN neurons, which each have only one value placeholder (to store the temporary result of the only differentiable arithmetic operation they are performing), f Net computational units have multiple value placeholders that are used by the suite of Net computational unit microprograms (containing both arithmetic and logical operations) embedded in the f Net computational units. These value placeholders are hereinafter referred to as switches and are described herein.

[0186]

[0187]

number

[0188]

number

[0189]

number

[0190]

number

[0191]

number

[0192]

number

[0193]

number

[0194]

number

[0195]

number

[0196]

[0197]

number

[0198]

number

[0199]

number

[0200]

number

[0201]

number

[0202]

number

[0203]

number

[0204]

number

[0205]

number

[0206]

[0207]

number

[0208]

number

[0209]

number

[0210]

number

[0211]

number

[0212]

number

[0213]

number

[0214]

number

[0215]

number

[0216]

number

[0217]

[0218]

number

[0219]

number

[0220]

number

[0221]

number

[0222]

number

[0223]

number

[0224]

number

[0225]

number

[0226]

number

[0227]

number

[0228]

number

[0229]

number

[0230]

number

[0231]

number

[0232]

number

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237]

number

[0238]

number

[0239]

number

[0240]

number

[0241] Units in each area (1)

[0242]

number

[0243]

number

[0244]

number

[0245]

number

[0246]

number

[0247]

number

[0248]

number

[0249]

number

[0250]

number

[0251]

number

[0252] The role of the f Net Compute Unit Switch is to help the f Net continually learn and control its world model, as described in Sections 3.2 and 3.3. Briefly:

[0253]

number

[0254]

number

[0255]

number

[0256]

number

[0257] 2 Training the model. At a high level, the primary role of an f-Net is to efficiently learn its world model: a compressed yet explainable encoding of the input sequence of f-Sim actions and observations, with the following explanation:

[0258] Learning efficiency. The efficiency of learning is This is seen as the ability to (i) learn a model of the Net input sequence from very sparse domain data, (ii) continuously extend this learned model (at test time) to capture the complexity of the real-world domain (rather than re-learning the model from scratch), and (iii) rapidly update the learned model (at test time) to rapidly adapt to non-stationarity in the domain. What enables f Net to have these desired characteristics is that the f Net connectom (unlike the ANN connectom) is continuously evolving (at test time) and consists of only those connections and computational units that were actually used for the task at hand. Furthermore, f Net computational units (unlike the ANN neurons) are equipped with the aforementioned switches, a kind of persistent memory that facilitates one-shot model updates (at test time).

[0259] Compressed encoding. Compressive coding is considered as the ability of an f-Net to encode all its input sequences using the smallest possible memory footprint. The measure of compression here is the inverse of the number of computational units the net is assigned to encode its input sequences. The compression itself may be lossless, measured as the accuracy of sequence continuation prediction and sequence recall, or it may not be lossless. Thus, CCE maintains a set of f-Net modules ranging from f-Nets that learn fast and provide lossless compression but have a large memory footprint (and require constant memory recycling) to f-Nets that learn slower and provide lossless compression but have a much smaller memory footprint and generalize better.

[0260] Explainable encoding. Explainable encoding is viewed as the ability of f Net internal (latent) sequence representations to be suitable for human inspection and validation. this is, (i) Discrete connectum coding, e.g., discrete horizontal connection weights

[0261]

number

[0262]

number

[0263]

number

[0264]

number

[0265] As shown in Figure 4, (unit

[0266]

number

[0267] 3.2.1 Information Flow. Prior to the detailed description of how fNet learns, compresses, disentangles, and predicts sequences, Fig. 4 and Fig. 5 provide a schematic diagram of the information flow during fNet model training. At a glance, the fNet model training algorithm uses four functions: ENTER() (Algorithm 1, see Fig. 24A), PARSE() (Algorithm 2, see Fig. 24B), ALERT() (Algorithm 4, see Fig. 24D), and PREDICT() (Algorithm 5, see Fig. 24E), which will be explained in more detail in later sections. unit

[0268]

number

[0269]

number

[0270]

number

[0271]

number

[0272]

number

[0273]

number

[0274]

number

[0275]

number

[0276] Referring now to Figure 5, this shows the flow of information into the f Net to prepare it for the arrival of some upcoming f Sim observation. To this end, the f Net ALERTS are the units that are likely to be activated next (after a new element is entered in the future). At a high level, this is accomplished as follows: Each currently activated unit broadcasts an ALERT message to units with which it has direct horizontal connections, and these alerted units then recursively pass the alert to the units below them, until the alert reaches a unit in region (1). For example, in FIG.

[0277]

number

[0278]

number

[0279]

number

[0280]

number

[0281]

number

[0282]

number

[0283]

number

[0284]

number

[0285]

number

[0286]

number

[0287]

number

[0288]

number

[0289] 3.2.2 Sequence learning and compression. Here, sequence learning is considered as the ability to predict the continuation of a sequence given some prefixes (context) of the sequence. One way to achieve this is to use n-grams or skip CTS, but such approaches are inefficient because they require encoding all possible 2-grams, 3-grams, etc. Another way to achieve this is to use autoregressive models, such as ANN encoder-decoder architectures. Such approaches can be highly compressible, but are data inefficient and only work in the stationary domain (when sequences are sampled from the same distribution).

[0290] A better way to achieve data-efficient compressed sequence learning is to use a hierarchical sequence encoding, e.g., the byte-pair compression / sequitur algorithm. At a conceptual level, f-Net works similarly to byte-pair compression: it assigns its computational units (recursively) only to the most common sequences of unit activations from the domain below, which correspond to the most common subsequences of unit activations from the domain below, and so on. What distinguishes f-Net sequence compression from byte-pair compression is that (i) f-Net performs sequence compression online, i.e., without access to a training dataset in advance; (ii) f-Net operates in a never-ending manner, i.e., without hard limits on the length or complexity of the patterns it encodes; (iii) this emphasizes the recency bias in its internal sequence encoding, thus enabling near-perfect sequence recall in nonstationary domains; and finally, (iv) it allows for much improved sequence compression by simultaneously separating seemingly unrelated subsequences that make up the input sequence of elements. That is, CCE takes into account the crucial observation that AGI agents will almost certainly be constantly exposed to sequences of heterogeneous / irrelevant elements (e.g., when distracted or multitasking between simultaneously pursued goals), and thus CCE f-Nets can seamlessly undo irrelevant subsequences of their input elements.

[0291] As described herein, we first show how the net learns and compresses discrete sequences using a fractal-like deep latent structure of computational units. We then show how the net internally represents intersecting sequences by reusing parts of previously learned sequences.Finally, in Section 3.2.3, it is shown how the net prepares for sequence interruption and resumption in order to achieve the desired sequence disentanglement performance.

[0292] Net computation unit learning threshold to start explaining sequence learning and compression algorithms

[0293]

number

[0294]

number

[0295]

number

[0296]

number

[0297]

number

[0298]

number

[0299]

number

[0300]

number

[0301]

number

[0302]

number

[0303]

number

[0304] For a first illustrative example of the sequence learning and compression process, see Figure 6. The fNet in Figure 6 has the following for all units:

[0305]

number

[0306]

number

[0307]

number

[0308]

number

[0309]

number

[0310]

number

[0311]

number

[0312]

number

[0313]

number

[0314] For a second illustrative example of the sequence learning and compression process, see FIGS. 7A-7C. Where:

[0315]

number

[0316] For yet another illustrative example of the sequence learning and compression process, see FIGS. 8A-8B, in which the same slow-f Net as FIGS. 7A-7C is used. This f Net has previously been exposed to a training corpus consisting of the sequences abcd and xyzv (sampled independently after an f Net attention reset), and is assumed to have perfectly learned an internal representation of these sequences. This example shows what happens when this f Net is now exposed to a new training corpus consisting of the sequence abcdxyzv. During the first pass, after the symbol abcd is processed at time 4, the network state (red attention box) is identical to the network state at time 24 from Figure 7C. Then, at time 5, the symbol x arrives, as shown in Figure 8A,

[0317]

number

[0318]

number

[0319]

number

[0320]

number

[0321]

number

[0322]

number

[0323]

number

[0324]

number

[0325]

number

[0326]

number

[0327]

number

[0328]

number

[0329]

number

[0330]

number

[0331]

number

[0332]

number

[0333]

number

[0334]

number

[0335] The example in Figures 8A-8B is somewhat idealized in that the underlying f Net is assumed to have first learned internal models for sequences abcd and xyzv (separately) and only after being exposed to the sequence abcdxyzv. In a more realistic scenario, the f Net would have learned representations for the sequences simultaneously, and certainly there would be portions of (seemingly unrelated) sequences that follow each other (in the dataset) and would be internally encoded and learned. For example, the top left diagram in Figure 9A shows a more realistic internal representation of an f Net about to begin learning the abcdxyzv sequence. Here, both internal representations for abcd and xyzv have already been learned, but in addition, the f Net has also fully learned representations for subsequences dx, cdx. The f Net would also learn the internal representations for abcd and xyzv, respectively.

[0336]

number

[0337]

number

[0338] Learning abcdxyzv, now an f Net, if exposed twice to abcdxyzv-abcdxyzv, will compress this longer sequence as shown in Figure 10 (note the appearance of fractal-like time-dilution gaps). More generally, slow-f Nets that continuously learn in real-world environments will typically have their internal sequence representations resemble fractals of time-dilution gaps, as conceptually shown in Figure 11.

[0339] Finally, in Figure 12, note how multiple learned sequences may reuse parts of the internal sequence representation that have already been learned. In this example, the sequence abijcd is learned first, followed by the learning of the sequence xyijzv. Now, we see how both sequences share a common internal representation of the subsequence xy (the units

[0340]

number

[0341]

number

[0342]

number

[0343] 3.2.3 Concurrency and Disentanglement of Sequences. Thus far, we have described how, in some examples, the f Net represents and tracks the progress of only one sequence at a time. In general, however, an f Net can track the progress of multiple sequences concurrently by alternating the sequence to which it currently attends (similar to how a CPU core handles multiple processes simultaneously by alternating between them). As described herein, this also allows an f Net to robustly attend to unexpected sequence interruptions and resumptions, a key feature that enables the underlying CCE execution agent to multitask among its domain action plans.

[0344] Specifically, one of the central features of human intelligence is the ability to seamlessly track the progress of multiple simultaneous events at different time scales. This is especially noticeable when these events correspond to action plans whose outcomes arrive delayed in time, and the underlying agent may not remain idle while waiting for these outcomes. For example, when remotely ordering lunch for residential delivery, the agent should not remain idle waiting for the lunch to arrive. Instead, it needs to proactively interrupt the execution of its current plan (e.g., eating a meal) and switch to some other plan that can be pursued in the meantime (e.g., reading a book). When the ordered lunch arrives, the agent needs to interrupt the currently executing plan of reading a book and resume the execution of the plan of eating a meal, exactly where it was interrupted earlier. Similarly, when the correct time arrives, the agent needs to know how to resume reading the plan of reading a book from exactly the point where it was switched off. Here we show how f-Net achieves that.

[0345] The process of disentangling f-Net concurrent sequences is

[0346]

number

[0347]

number

[0348]

number

[0349]

number

[0350]

number

[0351]

number

[0352]

number

[0353]

number

[0354]

number

[0355]

number

[0356]

number

[0357]

number

[0358]

number

[0359]

number

[0360]

number

[0361] To illustrate the preparation of the sequence interruption step in an example, please refer to FIG. Here, the f Net has already processed the sequence of elements xyij, at which point the sequence can proceed to either zv or cd. At this point, the f Net Two things can be done: (i) prepare the sequence for possible compression, and (ii) prepare the sequence for possible interruption.

[0362]

number

[0363]

number

[0364]

number

[0365] So, in the example of Figure 13 (focus on the filled circle in the upper left corner of the unit), Net

[0366]

number

[0367]

number

[0368]

number

[0369]

number

[0370]

number

[0371]

number

[0372]

number

[0373]

number

[0374]

number

[0375]

number

[0376]

number

[0377]

number

[0378]

number

[0379]

number

[0380]

number

[0381]

number

[0382]

number

[0383] The last switch

[0384]

number

[0385]

number

[0386]

number

[0387]

number

[0388] At time 1, element “a” is input, and the f Net calculates the number of units currently attending to it.

[0389]

number

[0390]

number

[0391]

number

[0392]

number

[0393]

number

[0394]

number

[0395]

number

[0396]

number

[0397]

number

[0398]

number

[0399]

number

[0400]

number

[0401]

number

[0402]

number

[0403]

number

[0404]

number

[0405]

number

[0406] The situation is that at time 3, instead of the currently likely element "c", the f Net unexpectedly receives element x at its input, complicating itself. The first consequence of this is that horizontal connections emerge between the previously attended and currently attended units in r1.

[0407]

[0408]

number

[0409]

number

[0410]

number

[0411]

number

[0412]

number

[0413]

number

[0414]

number

[0415]

number

[0416] The arrival of element "y" at time 4 is expected (probable) in region r1. There are strong connections from previous units in r1 to the currently attended unit.

[0417]

number

[0418]

number

[0419]

number

[0420]

number

[0421]

number

[0422]

number

[0423]

number

[0424]

number

[0425]

number

[0426]

number

[0427] The arrival of element "c" at time 5 is unexpected (impossible), and f Net would rather expect to receive "z" as input (which would advance the sequence xy that f Net was interested in), resulting in

[0428]

[0429]

number

[0430]

number

[0431]

number

[0432]

number

[0433]

number

[0434]

number

[0435]

number

[0436]

number

[0437]

number

[0438]

number

[0439]

[0440]

number

[0441]

number

[0442]

number

[0443] The arrival of element "d" at time 6 has a more severe impact.

[0444]

number

[0445]

number

[0446]

number

[0447]

number

[0448]

[0449]

number

[0450]

[0451]

number

[0452]

number

[0453]

number

[0454]

number

[0455]

number

[0456]

[0457]

number

[0458]

number

[0459]

number

[0460]

number

[0461]

number

[0462] Therefore, the arrival of "z" at time 7 is unexpected and therefore a new horizontal connection

[0463]

number

[0464]

number

[0465]

number

[0466]

number

[0467]

number

[0468]

number

[0469]

number

[0470]

number

[0471]

number

[0472]

number

[0473]

number

[0474]

number

[0475]

number

[0476]

number

[0477]

number

[0478] 3.2.4 Sequence Continuation Prediction. Thus far, it has been shown how the f-Net continually grows and updates its model of the world from intertwined sequences of f-Sim actions and the observations to which the f-Net attends. This learned world model is then used by the net to make the most likely predictions, across multiple time scales, about how these sequences will continue in the future.

[0479] To get an intuition about how the f-Net sequence continuation prediction algorithm works, consider the example in Figures 15A-15B, where f-Net has already built an internal representation of the sequence ijcd and a partial internal representation of the sequence xyijzv (in part,

[0480]

number

[0481]

number

[0482]

number

[0483] f Net unit switch

[0484]

number

[0485]

number

[0486]

number

[0487]

[0488]

number

[0489]

number

[0490]

number

[0491]

number

[0492]

number

[0493]

number

[0494]

number

[0495]

number

[0496]

number

[0497]

number

[0498]

number

[0499]

number

[0500]

number

[0501]

number

[0502]

number

[0503]

number

[0504]

number

[0505]

number

[0506]

number

[0507]

number

[0508]

number

[0509] In general, the use of fNet to predict the continuation of a currently attended sequence into the more distant future is

[0510]

number

[0511]

number

[0512]

number

[0513]

number

[0514]

number

[0515]

number

[0516]

number

[0517]

number

[0518] Thus far, Sections 3.2.2, 3.2.3 and 3.2.4 have introduced the core f-Net functions required to learn a world model. Once we have a good intuition for how these functions should work, the formal details of how these functions are implemented in a CCE are disclosed elsewhere in this document, e.g., Algorithms 1, 2, 3, 4 and 5 in Section 3.4. For the use of a CCE agent to control the world model, see Section 3.3 on learning how the f-Net uses its learned model to plan its next actions / options.

[0519] 3.3 Model-Based Policy Search. Having access to a learned world model provides an agent with a distinct advantage over model-free deep RL algorithms for planning its actions. Most notably, being able to predict the dynamics of the world allows for much improved data efficiency (fewer agent interactions with the world) for optimal control, better transition to other goals and tasks, and ultimately higher confidence in the underlying policy through model-based estimation of policy uncertainty. f-Net has already been demonstrated in practice to be able to continuously learn and update a model of the world from entangled sequences of heterogeneous f-Sim observations and actions. Here it is shown how this learned model is used by f-Net to plan future courses of action.

[0520] This document outlines four different algorithms that can be used for policy search: (i) Exhaustive Action Policy Search (with fixed horizon T), which treats the underlying f-Net model as a black box; (ii) Selective Action Policy Search (with infinite horizon), which exploits the inherent hierarchical encoding of f-Net sequences; (iii) Selective Option Policy Search, which exploits the inherent hierarchical encoding of f-Net sequences, but more strategically searches from a starting point where the above options have been tried, in the space of next options (macro-sequences of actions and observations) rather than the next action to be taken; and finally (iv) Exhaustive Option Policy Search, which extends option-based policy search to any starting point. See Figures 16A-16D for an explanation of the conceptual differences between the first three of these algorithms. 3.3.1 Exhaustive Action Policy Search (Horizon T) The first proposed algorithm for exploring action policies considers the learned model as a black box and considers all possible future action sequences of a given length T. That is, the planning agent does not look at the internal encoding of the f-Net sequence, but only needs to interact with the model by calling the ENTER() and PREDICT-ATTENTION-MOVES() functions (Algorithms 1, 5) to input observations (or actions) to the model and predict the next observation. In particular, for each unit

[0521]

number

[0522]

number

[0523] Formally, for a given time horizon T, the algorithm computes all possible action policies

[0524]

number

[0525]

number

[0526]

number

[0527]

number

[0528]

number

[0529]

number

[0530]

number

[0531] The above algorithm for exhaustive action policy search has two notable drawbacks: (i) it only works for a fixed planning horizon T, which is often small and chosen in an ad-hoc manner, and (ii) the set of possible policies Π T grows exponentially in T, and therefore has exponential complexity. While the effects of these drawbacks are less pronounced in short episodic domains (often considered in RL benchmarks), they are likely catastrophic in never-ending domains, where the planning agent can strategically consider action policies over a large (often unknown) time horizon. For example, in Figure 16, each of the 27 plausible action policies (each tree branch corresponds to some action and is eventually observed at some decision point) only considers the rewards collected at decision point 1, 2, 3 = T, and ignores all rewards at subsequent decision points T1, ..., T∞ connected via dotted lines. And because AGI agents will eventually operate in such open-ended domains, there is an urgent need for model-based planning algorithms that take such large time horizons into account, as we propose next.

[0532] 3.3.2 Selective Action Policy Search (Horizon ∞). A selective action policy search algorithm is proposed to specifically address the problem of large planning horizons. It is called selective because it does not consider all action policies of length T∞ for evaluation. Instead, it narrows its search space to policies that follow the most recently executed policy from some future Net state achieved at decision point T1′. The rationale for this heuristic is that once an agent masters a subpolicy from the fNet state achieved at decision point T′, it will insist on executing that subpolicy whenever it enters that state.

[0533] Thus, at decision point 1, the agent selects an initial action sequence of length T'-1.

[0534]

number

[0535]

number

[0536] See Algorithm 6 (see Figure 24F) for general pseudocode for Selective Action Policy Search. To illustrate one example of this algorithm, consider the third policy search tree in Figure 16C. Now, since T'=3 and there are three actions to choose from at each decision point, the agent considers 3(T'-1)=9 possible action policies of length T'-1. Each such course of action is assumed to be predictable at decision epochs T',...,T∞, in that at decision epoch T', given some starting state A(r),r=1,...R at decision epoch 1, it follows the most recently executed subpolicy from the f-Net state. For example, given the starting state of the f-Net at decision point 1 and the subsequent execution of actions at "left branch" and "middle branch", the f-Net predicts that at decision point 3, the action at "left branch" will be executed, and at decision point 4, the action at "right branch" will be executed. Note that this particular sub-policy rollout (marked with a dashed line) is different from the other rollouts at decision point T'. Also, although not shown, note that this exact rollout depends on what state A(r),r=1,...R f Net was in at the first decision point.

[0537] The actual value of T' may not be static. Depending on how long the receiving underlying agent takes to explore the policy at decision point 1, it may choose a larger value of T if that time allows, or a smaller value of T if a faster reaction is desired. Similarly, in an infinite horizon domain, an agent may choose to truncate its policy rollout horizon T∞ at will, or use a lenient discount factor to gracefully discount rewards from very far into the future, whenever a fast agent response is warranted. The receiving rollout itself is performed with EnterMode=Imaginary (see Algorithm 1, Figure 24A), so it does not result in connector growth, is fast (computational complexity O(R*T') in T' for a bounded number R of the fNet domain), and may not call the computationally expensive ALERT-LIKELY-UNITS() function in Algorithm 1 (see Figure 24A). And thanks to a specific internal encoding of the f Net sequence, when the states visited by the rollout start to form a cycle (for a certain unit

[0538]

number

[0539] The selective action policy search can be further improved retrospectively. As will be explained in detail in Section 5.4, the execution agent can retrospectively reconsider actions it has taken in the past and internally imagine having taken a different action at the corresponding past decision point in light of new evidence (e.g., discrepancies between the predicted and actually observed effects of performing these past actions). The advantage of using such retrospective reasoning is that future rollouts (performed by the selective action policy search algorithm at decision point T') are more likely to follow sub-policies with higher utility, thus resulting in a better overall policy at decision point 1. This may be particularly useful, for example, when an execution agent employing an ε-greedy search strategy wants to quickly forget about the random detouring action it took in the recent past while following an action plan with high utility.

[0540] 3.3.3 Selective Selection Policy Search (Horizon ∞). Model-based policy exploration theoretically offers significantly improved data efficiency, policy explainability, and policy transfer. In practice, the policies found by the underlying planning agents are often suboptimal, caused by inaccuracies in the learned world model, as well as computational limitations included in the planning process itself. These computational limitations are known to researchers in the planning community, and proposed mitigation strategies typically involve either (i) fixing the planning horizon to some small number in order to perform an exhaustive policy exploration (possibly improved by an iterative policy tree construction schedule and policy rollout), or (ii) allowing infinite planning horizons but using reward discounting factors (that discount rewards collected later in time) in order to find a fixed-point solution of the underlying Bellman equations. While these strategies have enabled impressive progress with the underlying agents in academic domains, their application to real-world, mission-critical domains has been delayed due to limited confidence in the found suboptimal policies.

[0541] At some point, the idea emerged to deepen the planning horizon while keeping computational overhead low by employing options in the planning process. Options are packaged, fixed sequences of actions and observations that an agent can consider pursuing (instead of or in addition to domain actions). For example, in the bottom policy tree of Figure 16D, in the first decision epoch, the agent can choose to pursue one of four permissible options: option 1, which allows the agent to consider an action policy: "left branch" → "middle branch" → "right branch" followed by a short rollout, which was originally missing from the set of selective action policies (in the third policy tree of Figure 16C); option 1, option 2 (green), which spans the entire domain time horizon T∞ and thus does not require any policy rollout; option 2, option 3 (pink) and option 4 (blue), which are relatively short and therefore potentially easier to be reused / transferred to several other decision points; In particular, note that option 4 (corresponding to the execution of just one action) is essentially a prefix of the larger option 3 when unpacked. It is also noted that the agent never considers the option of starting at the first decision point with a "middle branch" action, presumably because that action was never beneficial to the agent (at the first decision point) and has not been pursued frequently enough for the underlying option (starting with that action) to be learned.

[0542] The advantage of searching option policies over action policies is a significant reduction in the complexity of the search process. While the computational complexity of exhaustive action policy search in Figures 16A-16D is exponential in T∞ (e.g., there are 27 distinct action policies for T'=5), the computational complexity of selective option policy search is typically much lower (e.g., there are only four options to consider in Figures 16A-16D), provided the set of domain options is properly constructed. A natural question then becomes: how to determine whether an option is acceptable (and feasible) at a given decision point, and how to properly construct the set of domain options in the first place? Then, how is this achieved for a domain whose dynamics are represented by fNets?

[0543] Without any domain expertise, the set of domain options may not be handcrafted, but instead may be discovered automatically. Indeed, the topic of automatic option discovery has attracted a lot of attention, but only limited progress has been achieved so far, mainly on identifying first-level options. In this context, the method presented below for automatically discovering options and identifying their feasibility radically changes this situation. What makes this possible is the inherent encoding of the domain model using fractal networks, something that was missing in past approaches that assumed a black-box version of the world model. Specifically, it is the fractal network connectome growth that prioritizes the encoding of the most frequently encountered sequences (of domain actions / observations), which naturally leads to the discovery of higher level options associated with higher domain fNet units. And it is the persistent but rapidly modifiable fNet units that allow to almost instantly identify whether a given acceptable option is feasible in a given context (the current state of the fNet model).

[0544]

number

[0545] Formally,

[0546]

number

[0547]

number

[0548]

number

[0549]

number

[0550]

number

[0551]

number

[0552]

number

[0553]

number

[0554]

number

[0555]

number

[0556]

number

[0557]

number

[0558]

number

[0559] option

[0560]

number

[0561]

number

[0562]

number

[0563]

number

[0564]

number

[0565]

number

[0566]

number

[0567]

number

[0568]

number

[0569]

number

[0570]

number

[0571]

number

[0572]

number

[0573]

number

[0574]

number

[0575]

number

[0576]

number

[0577]

number

[0578]

number

[0579]

number

[0580]

number

[0581]

number

[0582]

number

[0583]

number

[0584]

number

[0585]

number

[0586]

number

[0587]

number

[0588]

number

[0589]

number

[0590]

number

[0591]

number

[0592]

number

[0593] Of the currently admissible options, only a few are currently feasible given the current state of the net. This means that the options pursued correspond to a sequence of fSim actions and observations, unlike the former (which are under the control of the agent), where the latter are simply a function of the current fNet state and its

[0594]

number

[0595]

number

[0596]

number

[0597]

number

[0598]

number

[0599]

number

[0600]

number

[0601]

number

[0602]

number

[0603]

number

[0604]

number

[0605]

number

[0606]

number

[0607]

number

[0608]

number

[0609]

number

[0610]

number

[0611]

number

[0612]

number

[0613]

number

[0614]

number

[0615]

number

[0616] To illustrate the search for option policies in an example, consider the f-Net in Figure 17. Having last observed o2, the f-Net is in states A(1)=4, A(2)=3, A(3)=2, A(4)=1, from which the underlying agents can either: (i) (to receive observation o6) negative

[0617]

number

[0618]

number

[0619]

number

[0620]

number

[0621]

number

[0622]

number

[0623] The proposed selective option policy search algorithm addresses this issue as it considers option policies that potentially adopt different actions at any decision point of the policy (relative to only those predicted by the model during policy rollout). This is best illustrated by revisiting the example in Figure 17. From the current state A(1)=4, A(2)=3, A(3)=2, A(4)=1, the algorithm can determine the acceptable options

[0624]

number

[0625]

number

[0626]

number

[0627]

number

[0628]

number

[0629]

number

[0630]

number

[0631]

number

[0632]

number

[0633] The last acceptable option to be considered

[0634]

number

[0635]

number

[0636]

number

[0637]

number

[0638]

number

[0639]

number

[0640]

number

[0641]

number

[0642]

number

[0643]

number

[0644]

number

[0645]

number

[0646]

number

[0647]

number

[0648]

number

[0649]

number

[0650]

number

[0651]

number

[0652]

number

[0653]

number

[0654]

number

[0655]

number

[0656]

number

[0657]

number

[0658]

number

[0659]

number

[0660]

number

[0661]

number

[0662]

number

[0663]

number

[0664]

number

[0665]

number

[0666]

number

[0667]

number

[0668]

number

[0669]

number

[0670]

number

[0671]

number

[0672]

number

[0673] We now conclude the description of the core features of the fractal network used by the underlying CCE executing agent to help it achieve its domain goals. As explained in Sections 5.4 and 5.4, the agent can achieve even higher performance if it additionally uses retrospective and counterfactual control. Briefly, retrospective control allows the agent to improve selective action policy exploration by revisiting past decisions and imagining what it would have done differently in the past in light of new evidence, encoding it into the model. Counterfactual control, on the other hand, allows the agent to consider pursuing options that are currently not feasible, but which would be better served by other policies that are executed preemptively, given the current state of the model.

[0674]

number

[0675] 3.3.4 Exhaustive option policy search (horizon ∞). Recall that Selective Option Policy Search only considers the feasibility of permissible options, i.e., options that have already been exercised (at some point in the past) in a given context (fNet state). Specifically, given a region r and its currently attended units,

[0676]

number

[0677]

number

[0678]

number

[0679]

number

[0680]

number

[0681]

number

[0682]

number

[0683]

number

[0684]

number

[0685] This selectivity in acceptable options therefore brought two advantages and one disadvantage. The advantages are that (i) the search for options policies was much more constrained and therefore faster, since not all f-Net options were considered, and (ii) the execution outcomes of acceptable options were already encoded in the f-Net model in each context (f-Net state), and therefore more predictable and reliable (although arguably less exciting for a novelty-motivated planning agent). In contrast, the main disadvantage of selective options policy search is that it does not directly admit options that are currently feasible but have not actually been tried in the current context (f-Net state). Thus, some favorable options were excluded from the planning process, potentially depriving the agent of higher quality policies.

[0686] To remedy this shortcoming, the planning agent can initiate an exhaustive option policy search that works as follows. First, the agent can list all of its f-Net units as potential options to undertake. Then, the agent can quickly filter the list by removing from it those options that cannot be initiated by any of the permissible actions (or currently predicted observations in region 1). Finally, for each remaining option from the filtered list, the agent can check whether this option is feasible using Algorithm 8 (see Figure 24H), and if so, the option utility is calculated by adding the reward of the region 1 units that the option unpacks (using Algorithm 9, Figure 24I), and optionally adding the reward of the region 1 units predicted to be visited after the option is completed (using Algorithm 10, Figure 24J). From the option that yields the highest utility, the agent can extract the first action to be performed and finally start executing said action until the arrival of the next observation, at which the exhaustive option policy search process resumes.

[0687] Thus, the exhaustive options policy lookup differs from the selective options policy lookup (Algorithm 7, Figure 24G) only in the list of options to consider, which is established in lines 2 and 3 of Algorithm 7, Figure 24G. Algorithm 11 (see Figure 24K) shows in detail how the list of options for this exhaustive options policy search is established.

[0688] 3.4 Pseudocode. This section provides pseudocode that formalizes the fNet core functions first introduced in Sections 3.2.2, 3.2.3 (for sequence learning, compression, concurrency and disentanglement), Section 3.2.4 (for sequence duration prediction) and Section 3.3 (for model-based policy search).

[0689] Sequence learning, compression, concurrency, and disentanglement. First, when a new action or observation (identified by the discrete symbol dst1) arrives in the f Net, it is entered into f Net region r=1 using the function ENTER(dst1) of Algorithm 1 (see Figure 24A). From then on (while loop in lines 2-15), this triggers a cascade of attention shifts in regions r=1, 2, 3, ... as follows: Assume A(r)=src is the identifier of the unit to which attention is directed in region r (line 3) and dst is the identifier of the unit that has just been activated in region r (line 4). Attention A(r) is immediately shifted to dst (line 5),

[0690]

number

[0691]

number

[0692] When the PARSE(r,src,dst) function in Algorithm 2 (see Fig. 24B) is called, the f Net can be in one of two EnterModes Real or Imaginary. In the former case, the actions or observations that enter the f Net are Real, e.g., they have been generated by some f Sim, while in the latter case they have been imagined internally by the corresponding CCE agent. The difference between the two EnterModes is that the f Net unit switches can be modified in any EnterMode, while the F Net (horizontal or vertical) connections can be modified only when EnterMode=Real. (Intuitively, the CCE can learn new concepts only if they are actually experienced, but it can revisit and update already learned concepts even if they are only imagined.)

[0693] The first task of PARSE(r,src,dst) (when EnterMode is Real on the first line) is a horizontal connection.

[0694]

number

[0695]

number

[0696]

number

[0697]

number

[0698]

number

[0699]

number

[0700]

number

[0701]

number

[0702]

number

[0703]

number

[0704] The second task of PARSE(r,src,dst) (when EnterMode is Real and parsing is unlikely to occur, as in line 8) is

[0705]

number

[0706]

number

[0707]

number

[0708]

number

[0709]

number

[0710]

number

[0711]

number

[0712]

number

[0713]

number

[0714]

number

[0715] The final task of PARSE(r,src,dst) is to determine the number of units in r1 that need to be activated next when the f-Net parses the attention transfer A(r) from src to dst.

[0716]

number

[0717]

number

[0718]

number

[0719]

number

[0720]

number

[0721]

number

[0722]

number

[0723]

number

[0724]

number

[0725]

number

[0726]

number

[0727]

number

[0728]

number

[0729]

number

[0730] As shown in Algorithm 2 (see FIG. 24B), a new horizontal connection

[0731]

number

[0732]

number

[0733]

number

[0734]

number

[0735]

number

[0736]

number

[0737]

number

[0738]

number

[0739]

number

[0740]

number

[0741]

number

[0742]

number

[0743]

number

[0744]

number

[0745]

number

[0746]

number

[0747] When the ENTER() function completes its main loop, the new attention units jointly mark the current state of the sequence to which they are currently attending. These newly attended units then send alerts to the units that are most likely to attend next by using ALERT-LIKELY-units(). The function works as shown in Algorithm 4 (see Figure 24D):

[0748]

number

[0749]

number

[0750] First, the current

[0751]

number

[0752]

number

[0753]

number

[0754]

number

[0755]

number

[0756]

number

[0757]

number

[0758]

number

[0759]

number

[0760]

number

[0761] First, fNet will be a solution to some of the existing vertical interconnections.

[0762]

number

[0763]

number

[0764]

number

[0765]

number

[0766]

number

[0767]

number

[0768]

number

[0769]

number

[0770]

number

[0771]

number

[0772]

number

[0773] The presence of three nested loops in ALERT-LIKELY-UNITS() potentially constitutes a computational bottleneck for fNet sequence learning. On the other hand, the break condition statement (line 6)

[0774]

number

[0775]

number

[0776] Sequence continuation prediction. f Net Sequence continuation prediction is performed using the PREDICT-ATTENTION-MOVES() function in Algorithm 5 (see Figure 24E). This function updates a prediction p of where (non-null) attention A(r) will move next in region r ← R, R-1, ..., 1 (lines 2, 3). For a given region r, p is assigned by default to the last attended-to successor of A(r) (line 4). This default assignment is important for proper CCE function. However, at this point, the underlying histogram for 1

[0777]

number

[0778]

number

[0779]

number

[0780]

number

[0781]

number

[0782]

number

[0783]

number

[0784]

number

[0785]

number

[0786]

number

[0787]

number

[0788]

number

[0789] [Number] is set in advance to j’ (line 14).

[0790] Selective action policy search.

[0791] The selective action policy search introduced in Section 3.3.2 is detailed in Algorithm 6 (see Figure 24F). The algorithm starts in the state where the f Net is given the input states A(r), r = 1, 2,..., R, from which the utility Q(a) of all possible actions a is calculated. The algorithm iterates over these actions (sequentially or in parallel, depending on the CCE implementation), and thus, first (at line 1) backs up the current f Net state and the values of the f Net unit switches. The backup itself is soft, and for example, only the changes registered to the f Net model may need to be stored in a backup data structure for later rollback to the original model. The algorithm then sets EnterMode to Imaginary to prevent the f Net connector mum from expanding during policy search and resets all values of Q(a) (lines 2, 3). The actual loop over all selective action policies executed at lines 4 - 15 considers all possible initial action sub - policies (a(1), a(2), …, a(T’ - 1)), where a(t) is the action that the agent executes at decision point t. The effect of each such sub - policy is first simulated by the learned fNet model: for each decision point t < T’, the algorithm calls ENTER(a(t)) to internally PARSE() the simulated action a(t), and A((t)) is its current state A(r) and prediction

[0792]

number

[0793]

number

[0794] After simulating the entire policy (initial subpolicy, followed by rollouts), the agent stores the policy expectation in q. The sum of rewards [a(t)] + rewards [o(t)] received at decision points t = 1, 2, ... T∞ is (optionally) discounted by δt for 0 < δ < 1 (line 11). As we explain later (Sec. 5.4), the agent then (optionally, line 12) further increases the policy reward q by CFR(), the intrinsic counterfactual reward for changing the value of f (identified in Sec. 3.3.3 as important for the feasibility of several promising master-level policies). Finally, Q(a) is updated to be its existing value or the maximum of q (line 13), and the f Net state and switch changes are rolled back using the backup stored in line 1.

[0795] Selective choice policy exploration.

[0796] The selective option policy search introduced in Section 3.3.3 is formalized in Algorithm 7 (see Figure 24G) and uses the functions FEASIBLE(), UNPACK(), and ROLLOUT() defined in Algorithms 8, 9, and 10, respectively. The selective option policy search takes as input the current f-Net state, say A(r) for r=1, 2, ..., R, and returns the highest utility Q(a) of actions that are acceptable in the current f-Net state and correspond to the initial action of a previously attempted option. The algorithm starts by initializing all Q(a) to -∞ in line 1. It then iterates over some currently attended unit in region r, for r=1, 2, ..., R, to find the highest utility Q(a).

[0797]

number

[0798]

number

[0799] First (line 4), the algorithm determines the currently considered option in the context of the current fNet state.

[0800]

number

[0801]

number

[0802]

number

[0803]

number

[0804]

number

[0805]

number

[0806]

number

[0807]

number

[0808]

number

[0809] Viable options

[0810]

number

[0811]

number

[0812]

number

[0813]

number

[0814] option

[0815]

number

[0816]

number

[0817]

[0818]

number

[0819]

number

[0820]

[0821]

number

[0822]

number

[0823]

number

[0824]

number

[0825]

number

[0826]

number

[0827]

number

[0828]

number

[0829] First, the unit

[0830]

number

[0831]

number

[0832]

number

[0833]

number

[0834]

number

[0835]

number

[0836]

number

[0837]

number

[0838]

number

[0839]

number

[0840] function

[0841]

number

[0842]

number

[0843]

number

[0844]

number

[0845]

number

[0846]

number

[0847]

number

[0848]

number

[0849]

number

[0850]

number

[0851]

number

[0852]

number

[0853]

number

[0854] lastly,

[0855]

number

[0856]

number

[0857]

number

[0858]

number

[0859]

number

[0860]

number

[0861]

number

[0862] From the updated f Net state A(r'),r'=1,2,...,R, the function then performs policy rollout in exactly the same way as Selective Action Policy Search Algorithm 6 (see Figure 24F). That is, when calling PREDICT-ATTENTION-MOVES() to update the updated f Net predictions, the function performs a loop over decision epochs t←1,2,...,T∞ (lines 17-22). At each epoch t, ​​it takes the predicted action a(t) and inputs it into the f Net (automatically updating the f Net state and next state predictions), and then takes the predicted observation o(t) and inputs it into the f Net again (also automatically updating the f Net state and next state predictions). The rewards for performing these actions and receiving the corresponding observations are then summed and (optionally) discounted before being returned by the function (line 23).

[0863] Comprehensive options list. Finally, to perform an exhaustive option policy search (Section 3.3.4), the agent establishes an exhaustive list of options that are (at least theoretically) applicable in the current f-Net state (Algorithm 11, see Figure 24K), and can then simply proceed with the execution of Algorithm 7 (see Figure 24G) from line 4 onwards, while ensuring that the FEASIBLE() algorithm does not invoke the instruction in line 5 (i.e., A[r-1] ≠ src and

[0864]

number

[0865] 3.5 Garbage Collector. If the number of f Net units exceeds a user-selected threshold, the f Net garbage collector can be invoked to reduce the number of f Net units to a desired level. The garbage collector itself may be indifferent to the relative importance of the units or may take this importance into account when selecting which units to remove, as will be explained next.

[0866] 3.5.1 Unit-Importance Agnostic Garbage Collector. A garbage collector that is unit-independent will prioritize units that represent more complex patterns / longer sequences when selecting which units to remove. For this purpose, it will start removing units from the top region first before moving to the lower region. When unit u is removed, all horizontal and vertical connections involving this unit may be removed and all switches of the remaining units that were pointing to the unit to be removed may be reset. Unit u may then be safely removed from the Net connectorum and the memory occupied by unit u may be deallocated and recovered by the system hosting the CCE.

[0867] 3.5.2 Unit-Importance-Aware Garbage Collector. A garbage collector that takes unit relative importance into account may calculate a unit importance I(u) for each fNet unit u when choosing which unit to remove first. In general, if u cannot be activated while traversing any sequence that encounters a non-zero reward, its importance is 0. Otherwise, I(u) is set to ActivationTime(u). The units are then sorted according to their increasing importance, and the garbage collector removes them one by one, starting with the least important units.

[0868] unit

[0869]

number

[0870]

number

[0871]

number

[0872]

number

[0873] 0 Focusing Simulator The Focusing Simulator module serves as an intermediate layer between the Domain Engine and the CCE Execution Agent and its f Net. The module consists of individual focused simulators, called f Sims, each wrapping one of the input signal modalities generated by the Domain Engine. Each f Sim runs asynchronously and is responsible for (i) maintaining a compressed, focusable intermediate representation of its modality, (ii) updating this intermediate representation as necessary whenever the Domain Engine generates the next frame in its modality, (iii) reporting to the CCE Execution Agent whenever the intermediate representation changes in the vicinity of the currently focused representation node, (iv) refocusing the intermediate representation whenever requested by the CCE Execution Agent, and finally (v) sending any updates to the currently executed domain action to the Domain Engine.

[0874] Regarding (i), the f Sim intermediate representation is derived from the original quad-tree data structure for the compressed representation of the image (called out in Section 4.1). f Sim further extends the focusing head mechanism by introducing a focusing head mechanism that allows f Sim to focus on selected nodes of the underlying tree. This mechanism is under the full control of the CCE execution agent, which autonomously issues asynchronous commands to shift the focusing head when necessary. Finally, the f Sim intermediate representation introduces the notion of oracle observations. These are observations that correspond to nodes of the underlying tree that are near the focusing head and whose changes are perceptible to the CCE executing agent.

[0875] With regard to (ii), the CCE execution agent, f Sim and the domain engine are all separate processes that run according to their own internal clocks. In particular, the domain engine is assumed to run at a rate specified by a given number of frames per second (FPS) that can vary over time. The FPS number itself can range from relatively small (e.g., less than 60 for the visual modality) to relatively high (e.g., more than 8000 for the auditory modality). The f Sim is assumed to run at a much higher rate, high enough to allow it to receive new frames from the domain engine and encode them into the intermediate representation.

[0876] Regarding (iii), it is assumed that only a selected small number of changes to the f Sim internal representation made after receiving a new frame for the domain engine are perceptible to the underlying agent and, as a result, are reported to the CCE execution agent. That is, f Sim reports to the CCE execution agent only changes to nodes (of the tree in its internal representation) that are currently close to the node pointed to by the focusing head. These are called current oracle nodes, and provide the CCE execution agent with oracle observations. The advantage of using oracle observations is that the agent receives significantly more information (from f Sim) about the current frame than is contained only in the node pointed to by the focusing head. However, because this additional information is already taken into account, the agent does not overfit the current intermediate representation of the current frame.

[0877] Regarding (iv), at any point in time, the CCE execution agent can request that f Sim refocus its current focusing head by instructing f Sim where to move its focusing head (in the intermediate representation). Focusing movement commands obey the physical limitations of f Sim, and they allow direct jumps to any node in the representation, whereas they incrementally follow the branches of the f Sim intermediate representation underlying the current frame. Furthermore, each focusing command takes a small amount of time, assumed to be smaller than 1 / FPS, to allow enough time to process the new frame information generated for the current modality by the domain engine.

[0878]

number

[0879] Regarding (v), finally, the CCE execution agent may at any time request the f Sim to perform a given domain action (or, for a proprioceptive modality, request a change to the values ​​of some nodes of the f Sim intermediate representation for that modality). When this request is received by the f Sim, it is passed to the CCE domain engine, which integrates the requested domain action when generating the next and subsequent frames. The execution of the domain action continues until the domain engine generates a frame whose intermediate f Sim representation contains a perceptible change in at least one of the current oracle nodes. This change is then reported to the CCE execution agent, closing the action observation agent loop.

[0880] In the remainder of this section, we first describe the f Sim architecture in detail. We then present examples of how the f Sim intermediate representations change upon execution of an active sensing policy in the visual modality. This is followed by a demonstration of how the Sim tracks and encodes modality changes extending along the time dimension. Finally, we explain how the underlying CCE-executing agent actively uses f Sim to maintain its situational awareness of the agent's surroundings.

[0881] 1 Architecture. Each of the focusing simulator modules in FIG. 1 consists of a compressed intermediate representation tree, a focusing head, and an oracle node, as described in this section.

[0882] 4.1.1 Quad-Tree Compression The architecture of f Sim is derived from the Quad-Tree data structure for compressed representation of images. In some cases, the systems and methods described herein are similarly applied to oct-tree and kd-tree data structures, which have shared computational properties for CCE but differ in computational behavior for various domains / dimensionality. The original quad-tree image compression method recursively divides an image into four small sub-images, such that the average shade of each sub-image is the average shade of its own sub-image. The recursive division then terminates if this average shade is within a given error margin from the shade in each of the sub-images. For example, in FIG. 18, an 8×8 monochromatic image is compressed using the quad-tree method, with the original image shown on the left and the corresponding quad-tree shown on the right. The root node of the quad-tree corresponds to the entire field of view (8x8 pixels) and its value is the average of the values ​​of its four child nodes, discretized into one of three buckets: white, gray, and black. The children of the root node cover smaller fields of view (4x4 pixels) and correspond to the (1) top-left, (2) top-right, (3) bottom-right, and (4) bottom-left quadrants, respectively. Nodes are considered to be leaf nodes if their further division does not yield new shading information. For example, child node number 1 of the root node (shaded in white), when expanded, will result in all of its children being shaded in white as well, thus terminating its division. In contrast, child node number 3 of the root node (shaded in gray) needs to be expanded since the shades of its children are not uniformly gray. As a result, the number of leaf nodes of the quad-tree is 16, which is much less than the total number of pixels in the original image, 64, resulting in a compression ratio of 4. Dependent on the number of discrete shade buckets dividing the underlying shade space, the ratio is smaller with an increasing number of buckets and larger otherwise.

[0883] Quad-tree compression naturally extends to other kinds of modalities. For example, if there are multiple color channels, there may be multiple corresponding quad-trees, each compressing its assigned color channel into its color shade value. Alternatively, the method can be adapted to either 1D or 3D image compression by assuming either duo-trees or octa-trees, respectively, where each range in each dimension is divided into exactly two sub-ranges. There may also be situations where one would like to divide each dimensional range into more than two sub-ranges (e.g., if robustness or redundancy of the compressed data structure is required), or where the sub-ranges partially overlap. Alternatively, if the shade values ​​of all child nodes are all within a narrow sub-range, one may want to first normalize these ranges to a larger range and only then apply discretization to achieve better relative (among children of a node) shade discrimination, at the expense of reduced accuracy of absolute shade values. These are, among others, some possible extensions of the original quad-tree data structure that are relatively easy to implement and experiment with.

[0884] 4.1.2 f Sim module. The focusing simulator module derives from the original quad-tree compression method in that it also maintains a compressed, tree-like representation of the underlying frames produced by the domain engine. (We refer to this fSim representation from now on as the intermediate representation, since it sits between the original domain engine image representation and the internal fNet representation.) But importantly, fSim enables even better compression ratios by presenting to the CCE execution agent (and the underlying CCE fNet) only selected parts of that intermediate representation - those parts that are likely to be relevant to the policy the CCE agent is pursuing.

[0885] To achieve that, f Sim employs the concepts of focusing head and oracle observation, as shown in Figure 19. Essentially, f Sim allows the CCE agent to control f Sim's focusing head such that it probes only those parts of the f Sim intermediate representation (of the domain engine frame) that the CCE agent considers important for the execution of the current CCE agent's policy. For example, in the intermediate representation in Figure 19, the CCE agent simply issues a command to f Sim to move the f Sim focusing head from the root node at zoom level 1 along direction number 2 to reveal the contents of the top right quadrant of the original image. Once this movement of the f Sim focusing head is completed (solid red square), f Sim reports back to the CCE agent the current contents of that intermediate representation node that the focusing head points to ("patch-zoom2-gray"). Additionally, f Sim also informs the CCE agent about the current contents of its intermediate representation oracle nodes (encircled by dashed red squares), which are essentially the direct destinations to which the focusing head can move next. Thus, the CCE agent receives information from f Sim about the shading of only 9 of the total 21 f Sim intermediate representation nodes, resulting in an additional 2.33 compression ratio of the original image (or a total of 8.66). The drawback of the CCE agent is that it does not (yet) know at all what the exact contents of the bottom right (8x8 pixel wide) quadrant of the original image are, other than that there is something in that quadrant (due to the oracle observation "patch-zoom2-gray" at that node). Importantly, however, for a given policy currently being pursued by the CCE agent, the exact contents of that quadrant may be entirely irrelevant, resulting in notable benefits (less Net memory to store the image, less time to explore other parts of the image, and less risk of the CCE agent policy overfitting to irrelevant observations).If not, the CCE agent can command the f Sim to execute a sequence of focusing head movements, henceforth called a focusing policy. How this is accomplished for different kinds of f Sim modalities is described next.

[0886] 4.2 Modality. This subsection presents Sim focusing policies for the spatial modality, the temporal modality, and the most common time graph modality.

[0887] 4.2.1 Spatial modality. It is first demonstrated how the Sim focusing policy works for modalities that have spatial dimensions but no temporal dimension. Examples of such modalities are 1D images (e.g., representing real values ​​between 0 and 1), 2D images (e.g., representing photographs), or 3D images (e.g., representing depth maps). fSim handles such spatial modalities in exactly the same way, using an intermediate (quad-tree-like) representation, an attention head, and an oracle node, as described in Section 4.1.2. The only differences are the fSim parameters that dictate (i) the number of quadrants used for each dimension, (ii) the relative quadrant overlap, (iii) the selected number of buckets into which the shade values ​​are discretized, and (iv) whether to normalize the shade values ​​of the children of each node in the intermediate representation.

[0888] These parameters are best illustrated with an example. Recall the intermediate representation of Fig. 19, corresponding to a monochromatic visual modality of 8x8 pixels. Here, the intermediate representation has a relative quadrant overlap of 0%, considering only two quadrants per dimension used, resulting in exactly four child nodes of each node in the intermediate representation. For more efficient recognition of shapes on the boundaries between quadrants, an alternative is possible with a relative quadrant overlap of 50%, considering three quadrants per dimension, resulting in exactly nine child nodes of each node in the intermediate representation. Alternatively, one can consider four quadrants per dimension, with no quadrant overlap, aiming for a shallower intermediate representation tree (only three zoom levels), but with a higher branching factor of the tree, so that each focusing action reveals information covering a relatively small field of view.

[0889] The number of buckets into which the shading values ​​are discretized is also an fSim parameter that can be carefully chosen. A higher number of buckets naturally leads to a higher shading precision for the node, and therefore more accurate shading information is sent back to the CCE agent. However, this also potentially leads to over-fitting the CCE agent policy, resulting in overly precise shading variations, negatively impacting the generality of said policy. In some instances, a smaller number of buckets (e.g., three in Fig. 19) may be preferable, since the agent can always instruct fSim to zoom in on individual pixels on a smaller sub-image in order to reveal (and encode in the dithering pattern) the absolute shading, even at the expense of a longer focusing policy. Thus, the right balance between the generality of the focusing policy (by varying the number of discretization buckets) and its length is important for optimal performance of fSim.

[0890] Finally, fSim's design can employ normalization of node shading to increase the relative contrast between the shadings of a node's children. This can be achieved by computing the current range of shading values ​​for all child nodes at agent runtime, and then discretizing this (narrow) range into a selected (small) number of buckets. Individual shadings are then normalized by projecting them into that narrow discretized range. In addition to this technique allowing fSim to encode the finest details in the relative shading of nodes, it also allows the encoding of images that preserve color constancy - the tendency for objects to be perceived (by a human observer) as having the same color even when lighting changes.

[0891] Focusing policies for spatial modalities. Thus, the CCE agent encoding of (the f Sim intermediate representation of) objects in the spatial modality is a directed graph (with cycles) whose vertices are the discretized shade values ​​(of the nodes of the f Sim intermediate representation) combined with their corresponding zoom levels, and the graph edges are the focusing head commands issued to transition to these nodes. To illustrate this, consider as an example the focusing policy of Figs. 20A-20D. Here (see the first tree), the focusing head is first issued a "focuszoom1" command (by the CCE agent), which results in it moving to the root node (in the intermediate representation tree) and observing the shade "patch-zoom1-gray", which most accurately depicts the shade in the entire 8x8 pixel field of view of the original image. At that point, f Sim sends back to the CCE agent this shade "patch-zoom1-gray", as well as the shade of the current oracle node (marked with a red dashed box), which is a potential node to which the focusing head can move next. Thanks to this auxiliary information from the oracle node, the current perception of the CCE of the image (left image in the f Sim intermediate representation) is much richer, covering the entire field of view but also showing some of the shadow details in each of the quadrants at zoom level 2.

[0892] The second tree in Figure 20B is the intermediate fSim representation after the focusing head has been commanded to move from zoom level 1, along direction 2, to zoom level 2 (denoted as "focus-zoom1-dir2"). Here, the focusing head points to a node whose shading is "patch-zoom-2-gray" as depicted, but the oracle node reveals seven extra (oracle) observations from zoom levels 1, 2, and 3 in the intermediate representation tree. These are all passed to the CCE agent, whose current perception of the image (visualized on the left side of the tree) includes the shading patch "patch-zoom2-gray" that the focusing head is pointing to, the low-resolution peripheral vision outside the focusing heads red box (zoom 1 along directions 1, 3, 4), and the high-resolution vision of the inner visual patch that would be obtained if the focusing head were to execute a zoom in command from its current position. Importantly, the CCE agent not only learns (and memorizes in its fNet) the (attentionally directed) sequence of its executed focusing-head commands and focusing-head observations, but also (carefully) updates its existing fNet unit switches to store the most recent oracle observations in relation to the current position of its focusing head. This allows the CCE agent to persistently recognize these oracle observations even after they are no longer perceptible.

[0893] The third tree in Figure 20C is the intermediate fSim representation after the focus head has been commanded to move from the previous node at zoom level 2, along direction 3, to zoom level 3 (denoted as "focus-zoom2-dir3"). Here, the focusing head points to a node whose shading is "patch-zoom3-gray" as depicted, but the oracle node reveals 12 extra (oracle) observations from zoom levels 1, 2, 3, and 4 in the intermediate representation tree. These are all again passed to the CCE agent, whose current perception of the image is shown to the left of the fSim intermediate representation tree. Note in particular how the CCE agent accurately perceives pixel-level information about the shading of the original image, albeit in a narrow 2x2 field of view. And, despite focusing on such a narrow field of view, the CCE agent also perceives (eye) observations at zoom levels 1, 2, and 3 in its peripheral field of view.

[0894] Finally, the fourth tree in FIG. 20D shows an intermediate fSim representation after the focusing head has zoomed out to zoom level 1 and has been commanded to move the focusing head along direction 3 (denoted as “focus-zoom1-dir3”). Note how the detailed content in quadrant 2 at zoom level 2 has disappeared since the focusing head is no longer pointing in this direction. Instead, more details are revealed in quadrant 3 at zoom level 2, which is where the focusing head has moved and where new oracle observations are provided to the CCE agent. At this point, the CCE agent may decide to continue with the focusing policy and gather more information about the image, or to perform some other task if the gathered information is predicted to be sufficient for successful execution of the above task.

[0895] 21A-21B summarize the focusing policy executed by the CCE agent. As can be seen, it consists of only four focusing head movement commands, each followed by a corresponding observation of the patch shading at a given zoom level. Note how the representation (similar to convolutional ANNs) allows for object translation invariance: from a particular node in the intermediate representation, all subsequent sequences (of visual patches and focusing head commands that together identify an object) are invariant to the exact path from that node to the root node in the intermediate representation tree. But crucially (unlike convolutional ANNs), these subsequent sequences are observed (and stored) in the context of past sequences (e.g., encoding the relative position of the object with respect to other objects), ensuring that object recognition is context-related versus decoupled from any context (as is done by individual filters in a CNN). Moreover, the CCE agent representation is presumably more resilient to individual pixel adversarial perturbations, and the f Sim intermediate representation progressively filters out noise (or adversarial pixel perturbations) the closer the focusing head gets to the root node of the tree (and the more likely it is to be in the first place). Also note that (unlike convolutional ANNs) the f Sim intermediate representation is essentially invariant to image resolution and therefore future-proof. This allows the underlying domain engine to be upgraded over time, producing higher resolution images without the need to retrain the underlying CCE agent. (All the CCE agent has to do then is allow f Sim to start focusing on previously inaccessible deeper nodes in its intermediate representation.)

[0896] Thus, the actual focusing policy of the CCE agent is rather compact. Moreover, the cumulative image representation of the CCE agent is much richer, as shown in the left image of Fig. 21A. If all the information for the oracle observations is taken into account (note that this information is stored in the CCE agent's f-Nets in the form of 17 oracle observation sequences registered by the underlying f-Nets), the cumulative image representation of the CCE agent is remarkably close to the original image generated by the domain engine. And although the CCE agent does not perceive all of these observations simultaneously, it always recognizes them, because it can choose to internally predict the continuation of these recent observation sequences and thus imagine finer details of the image that are no longer perceptible. An interesting comparison can then be made between the focusing policy and the image recognition of the CCE agent, and between the saccadic policy and the image recognition observed in human vision. In Fig. 21B, the registered saccadic movements of the human eye (when looking at an image of someone's face) only allow very selective high-resolution probing of the underlying image parts. Furthermore, the accumulated saccadic information, combined with the human ability to imagine parts of the visual scene that are (no longer) perceived, gives the internal impression that the image is always perceived in high resolution.

[0897] In summary, therefore, the CCE agent representation of the spatial modality is a directed graph whose edges are f-Sim focusing head movements, while the vertices are the perceived visual patches. And the goal of the CCE agent focusing policy is to recognize objects by efficiently traversing only the relevant (for a given task) part of that graph. Such traversal, together with the underlying focusing head movement commands, is intended to reveal visual patches from the f-Sim observation sequence that have already been learned by the underlying focusing net and are predicted to lead to positive rewards accruing for the completion of the underlying domain task.

[0898] 4.2.2 Temporal modality. The real-world domain and the actions that agents perform, including f Sim focusing head commands, consume time. Therefore, any domain modality that a CCE agent perceives is explicitly extended along the time dimension. This section shows how such extensions of the time dimension are handled by CCE.

[0899] The main difference in CCE processing for temporally extended modalities is that some parts of the f Sim intermediate representation of a given modality (the shading of nodes within these parts) may change over time when the domain engine generates a new frame for that modality. Most of these changes may go unnoticed by the CCE agent, which sees only a small portion of the f Sim intermediate representation tree at any given time (via oracle observations). However, if the f Sim focusing head is near these changed nodes, the CCE agent will perceive the change in shading in the underlying oracle observations and potentially suspend the execution of its current policy. At that point, the agent autonomously chooses which node of the f Sim intermediate representation (the one whose shading change was perceived by the agent) to focus on next by commanding the f Sim focusing head to move towards that node.

[0900] Remarkably, such processing of the temporally extended modality is also encoded as a directed graph (whose edges are focusing head commands, while its vertices are patches of frames from the temporal modality), nearly identical to the CCE agent encoding of the spatial modality. Similarly, with a given task in mind, recognizing a pattern in such a temporal modality consists of traversing the correct path (sequence of observations) in that graph to reveal sufficient information for the task at hand. This nearly homogeneous CCE agent encoding of heterogeneous spatial and temporal modalities is what allows the agent to seamlessly blend its policies in these modalities. It allows the agent to effectively merge the underlying directed graphs and store it efficiently (in f-Net) as one policy graph. It then simultaneously issues commands to the focusing heads of the corresponding f-Sim modalities according to this combined policy graph.

[0901] Focusing policies for time modalities. To illustrate the focusing simulator and focusing policy for the time modality, see the example in Figures 22A-22E. The first diagram in Figure 22A shows the original pattern evolving over time (sampled on the x-axis). At any point in time, the pattern itself is a 1d monochromatic image consisting of eight non-overlapping value bands (e.g., a simplified spectrogram with eight frequency bands). The intermediate representation tree of the f Sim assigned to handle that modality has a branching factor of 2, meaning that the image can be split into two non-overlapping parts, each of which can itself be split into two non-overlapping parts, and so on. Thus, the f Sim focusing head can move either up or down from a given zoom level, denoted as "focus-z2-d1" and "focus-z2-d2" for zoom level 2, respectively. The shading pattern of the intermediate representation nodes is now more complex (than a single shade) in that it consists of an upper and lower part; e.g., "patch-z1-wg" corresponds to a white (w) upper part and a gray (g) lower part at zoom level 1, while "patch-z3-bw" corresponds to a black (b) upper part and a white (w) lower part at zoom level 3, etc. The goal of a CCE agent is to learn to recognize spatiotemporal patterns.

[0902] The second diagram in FIG. 22B illustrates a first exemplary focusing policy of the CCE agent. The policy is extreme because it does not include any focusing head movement to zoom levels greater than 1, and is therefore likely to be among the first to be executed by the CCE agent. Specifically, the agent first (before time t1) commands the focusing head to focus at zoom level 1 (the “focus-z1” command) and observes the entire field of view (8×1 pixels), albeit at a low resolution. The shading pattern of the node to which the focusing head points is “patch-z1-wg”, and as seen in FIG. 22, this pattern does not change during time snapshots t1, t2, t3, t4. For ease of illustration, we now show oracle observations from the child nodes of the currently pointed-to node. During these times, the CCE agent may appear to be idle to an external observer. Nonetheless, not distracted by the eventually perceptible observation, the CCE agent can effectively engage in other activities, for example related to computationally expensive model-based policy exploration. This continues until time t5, when the domain engine generates a frame that eventually results in a change in the shading pattern from "patch-z1-wg" to "patch-z1-gg" (at the root node of the f Sim intermediate representation tree). This is immediately noticed by the CCE agent, which reissues the command "focus-z1" and receives the underlying pattern "patch-z1-gg" from f Sim. The agent's perception of the current pattern then remains unchanged until time t7, when a substantially different new frame from the domain engine similarly causes the agent to reissue the command "focus-z1" and observe the underlying pattern "patch-z1-bg". Finally, a subtle change in this pattern at time t8 is imperceptible to the agent, even though it is likely to have been registered at some more distant (away from the root) node of the f Sim intermediate representation tree.In summary, agent focusing policy 1 consists of only three focusing head movements and is therefore the coarsest grained (in terms of time scale), and it encodes some of the most salient features of the original spatiotemporal pattern (perceptible even at zoom level 1) and is likely to generalize well to other settings, although some tasks may require it to be further specialized.

[0903] One such specialization of a policy is shown in Focusing Policy 2 in Figure 22C. Here, the agent similarly (as in Focusing Policy 1) initially issues a "focus-z1" command and receives the observation "patch-z1-wg" at time t1. However, unlike Policy 1, instead of idling until t5, the agent (knowing that the bottom part of the image is not uniformly white or black and thus may contain some non-uniform pattern, not yet revealed) chooses to issue a "focus-z1-d2" command and learns from the corresponding observation (received from f Sim) that the bottom part of the original image contains a "patch-z2-bg" pattern. Note how this pattern does not change in all subsequent time snapshots and therefore does not distract the agent. However, one of f Sim's current oracle nodes, corresponding to the root node of the intermediate representation tree, changes its value from "patch-z1-wg" to "patch-z1-gg" at time t5. The agent is therefore distracted thereby, and is forced to issue the "focus-z1" command, observing the underlying pattern "patch-z1-gg". At this point, the agent could choose (as in policy 1) to remain idle until time t1, when the "patch-z1-gg" pattern changes to "patch-z1-bg". But instead, seeing that there is something to uncover at both the top and bottom of the image (which are grey in the "patch-z1-gg" pattern), it decides at time t6 to issue the command "zoom-z1-d1", focusing on the top, and receiving the pattern "patch-z2-bg". Immediately after, the agent is "double" distracted.Not only did the “patch-z2-bg” pattern (at the node the f Sim focusing head points to) change to “patch-z2-bb” (blue box), but the pattern “patch-z1-gg” (the oracle observation corresponding to the root node in the intermediate representation tree) simultaneously changed to “patch-z1-bg” (red box). As seen in Figure 22C, in this example, the agent chooses to focus on the latter observation (red box) and then remains idle at time t8. In summary, the agent’s focusing policy 2 consists of five focusing head actions, three of which match the actions from policy 1, and the remaining two provide the agent with additional details about the original spatiotemporal pattern. Note, however, that in order to gather this extra information, the agent was required to reduce its idling time from five to three ticks, resulting in less available time that could be allocated to performing, for example, model-based policy exploration.

[0904] Finally, at the opposite extreme, the agent can choose to use all idle time for spatiotemporal pattern recognition, as shown in Focusing Policy 3 in FIG. 22D. At time ticks t1 and t2, the policy matches Policy 2. But instead of idling at time t3, the agent chooses to completely disambiguate the shading pattern "patch-z2-bg" at the bottom of the image by issuing a "focus-z2-d2" command to receive the underlying pattern "patch-z3-wb". As the shading pattern at the node where the focusing head is currently pointing continues to fluctuate (between "patch-z3-wb" and "patch-z3-bw") in subsequent time ticks, the agent is busy trying to track it, repeatedly issuing the "focus-z2-d2" command to fSim. In fact, the agent is so preoccupied with tracking such small variations that it chooses to ignore the change in the oracle observation at the root node of the intermediate representation tree (from “patch-z1-wg” to “patch-z1-gg”) at time t5. However, at time t7, the agent finally issues the “focus-z1” command and receives the observation of “patch-z1-bg”, giving it the urgency to search for a significant change in the shading at the oracle node at the root node of the tree. In doing so, the agent then chooses to immediately (at time t8) refocus on the bottom of the image by issuing the command "focus-z1-d2" and receiving the observation "patch-z2-bg" rather than wasting time being idle. In summary, the agent's focusing policy 3 consists of eight focusing head movements, encompassing all possible zoom levels, and using all eight available time ticks to track the underlying spatiotemporal pattern. Visual inspection of the accumulated visual representation over time (including all received oracle observations) reveals that it partially resembles the original spatiotemporal pattern, but also offers a degree of generality.

[0905] Most importantly, the goal of the CCE agent focusing policy is not to reveal all the most frequent features of the original spatiotemporal pattern. Unlike the ANN agent, which captures all these features indiscriminately, the CCE agent focusing policy aims to reveal only those features that are relevant to the given domain task at hand. Thus, given different domain tasks, the CCE agent will most likely adopt different focusing strategies and will reveal slightly different parts of the original spatiotemporal pattern each time.

[0906] Consider the graph at the bottom of Fig. 22E, aggregating focusing policies 1, 2, and 3. Here, the red edges represent the focusing head commands that were issued, while the blue edges represent alternative focusing head commands that could have been issued but were not. The nodes of the graph represent the shading patterns observed by the focusing head at the time of the execution of the focusing head commands. The graph itself represents the learned perspective CCE agent representation of the original spatiotemporal pattern. (The actual graph is stored in a compressed form in the underlying CCE agent's fractal network.) Then, depending on the domain task at hand, the agent plans an optimal traversal of that graph, gathering all of the relevant observations to successfully complete the task with as little effort as possible. Along the way, the agent perceives many auxiliary observations that are provided to the agent via oracle observations, giving the impression that the agent perceives more observations than it is focused on. Moreover, it constantly performs its sequence continuation prediction along paths that virtually branch off from its main focusing policy path, further increasing its awareness of what is happening in the parts of the pattern that are currently out of focus. Finally, when following the main path, the main path proactively prepares itself for interruptions or policy bypasses of this main path, so that it can seamlessly resume following the main path in the future. For example, in running "focus-z1" and encountering "patch-z1-wg", it (agent f Net) prepares to encounter "patch-z1-wg" when the "focus-z1" command is issued again (on the main path). If it then encounters a policy bypass that starts along the "focus-z1-d1" edge, it will still be able to reconnect parts of the main path when it eventually perceives "patch-z1-gg" as one of its oracle observations.

[0907] 4.2.3 Graphical modalities. So far, it has been shown how a CCE agent can track the progression of spatiotemporal patterns through interfacing with the intermediate representation tree of the corresponding f-Sim. This section outlines an extension of this method to allow a CCE agent to track the temporal progression of modalities represented as directed acyclic graphs. This extension is motivated by the desire to allow CCE to be efficiently applied to dynamic network domains, as well as by demonstrating how a CCE f-Sim can handle somatosensory and proprioceptive modalities of the underlying AGI agent.

[0908] In general, a temporal graph modality is represented by a directed acyclic graph G = (V, E), where V is the set of vertices of the graph and E is the set of edges of the graph. The set V consists of source nodes with only outgoing edges.

[0909]

number

[0910]

number

[0911]

number

[0912]

number

[0913]

number

[0914]

number

[0915]

number

[0916]

number

[0917] As proposed in Section 4.2.1, the shadow aggregation function c(v) of a vertex v is

[0918]

number

[0919]

number

[0920]

number

[0921] In this case, the f Sim intermediate representation of the time-graph modality described above is no longer a tree, but the graph G = (V,E) itself. As a result, the focusing head no longer points to a tree node, but instead to a given graph vertex

[0922]

number

[0923]

number

[0924]

number

[0925]

number

[0926]

number

[0927]

number

[0928] 4.2.4 Controllable Modalities. A final note in the focusing simulator section is that of all the modalities that the CCE agent perceives, some are also agent-controllable (e.g., motor / sensory modalities). That is, for controllable f Sim modalities, the agent can not only perceive the current shade values ​​at the nodes of the f Sim intermediate representation of that modality, but can also spontaneously issue commands to the f Sim that change the current shades of these nodes to desired values. Note that each such CCE agent command will not affect just one pixel / sensory value of the underlying modality, but rather an entire subframe of pixel / sensory values, albeit at a lower resolution. To issue more sophisticated changes, the agent would then similarly issue a "zoom in" focusing command to the focusing head, followed by a shade-changing command to a smaller subframe, and so on. The notion of performing a "domain action" is then fundamentally different. Instead of performing atomic domain actions, the agent relies on f Sim to unpack these changes in its intermediate representation tree nodes into pixel / sensory values ​​of the underlying controllable modality and issues a sequence of focusing head movement and node shading commands to f Sim.

[0929] Thus, the CCE-executing agent simultaneously interacts with a set of f Sim modalities corresponding to all modalities generated by the domain engine. The agent simultaneously perceives oracle observations from all modalities and issues commands to the focusing heads of each modality in parallel. It learns a model of the world by growing its fNet connectome from sequences of heterogeneous f Sim actions and observations. It then uses this learned world model to find a goal-achieving action policy that it sends to the domain engine via the f Sim intermediate representation. How this is achieved in detail is explained next (Section 5).

[0930] 4.3 Pseudocode. Pseudocode for the focusing simulator is provided in the UniversalModality class in either the Python or C++ implementation of CCE. For a given f Sim modality, we write the following functions: *AdmissibleFocusingActions. This function returns a list of permissible focusing head movements from a given node in the f Sim intermediate representation (for the given modality) (e.g., the node the focusing head is currently pointing to). This is then automatically translated into a list of oracle nodes that are currently perceivable by the CCE agent. *PerceivableOracleObservations. Each fSim is set up to work with either "low", "medium" or "high" perceptual sensitivity settings, and specifies a subset of oracle nodes, a change in value of which will interrupt the currently executing action of the CCE agent. The "high" setting includes all current oracle nodes, the "medium" setting removes oracle nodes that require the focusing head to zoom in from the current node, and the "low" setting includes only those oracle nodes that are on the path from the current node to the root node in the intermediate representation tree. For the selected perceptual sensitivity setting, PerceivableOracleObservations returns the current subset of oracle nodes from the intermediate representation tree. *GetObservation: This function returns the average shading pattern at a given node of the intermediate representation tree. This function uses a well-known algorithm to compute (in constant time) the sum of all elements in a sub-matrix (which includes only voxels corresponding to sub-images of a given modality).

[0931] Please refer to the Python or C++CCE codebases of these features for their implementation details. 5.0 Execution Agent

[0932] This section describes different kinds of CCE Execution Agents, responsible for controlling the interaction between the CCE fractal network and the domain engine via Focusing Simulator intermediaries (recall Figure 1). What makes CCE Execution Agents distinctive is that they improve upon a pesky shortcoming of existing deep reinforcement learning agents: the need to synchronize the agent and the domain simulator. Moreover, CCE Execution Agents finally provide mechanisms for counterfactual planning and multitasking, which are possible features of future AGI agents.

[0933] To illustrate the essential differences between various CCE execution agents, please refer to Fig. 23. First, a synchronous non-focusing agent implements the most basic agent / domain engine interaction loop from the RL literature, where (1) the domain engine generates a frame (observation), sends it to the agent, and goes to sleep, (2) the agent wakes up in response to the just received observation and computes a policy, (3) the agent sends the first action from the found policy to the domain engine, then goes to sleep, and finally, (4) the domain engine receives an action from the agent, wakes up when it generates a new frame, etc. Synchronous non-focusing agents have two significant drawbacks: (i) when the domain engine generates frames at a high rate, the agent often does not have enough time (between frames) for policy exploration, regardless of how similar subsequent frames are to each other. This makes the agent act in a squashed manner, making the agent behave suboptimally; (ii) due to the high frame rate, model-based RL agents store very long sequences of action observations (even if subsequent frames are nearly identical), making policy representation and policy search incomputable. To mitigate these significant drawbacks, model-based RL approaches have been largely abandoned in favor of model-free approaches that do not attempt to learn and memorize the dynamics of the underlying domain. And model-free approaches (which attempt to encode the last few frames from the domain engine into the current state to compute higher-order Markov policies) resort to ad-hoc discretization of time that only considers every k-th frame from the domain engine. Also, Semi-Markov Decision Processes cannot address the current problem because they were proposed to handle uncertain frame arrival times and are not useful for modeling a high rate of frames arriving at uniform intervals from the domain engine.

[0934] The proposed CCE execution agent radically improves these shortcomings by gradually relaxing the interlocking of the agent and domain engine clocks and realizing a completely asynchronous execution of both processes. First, the synchronous focusing agent (see the middle diagram in Fig. 23) significantly reduces the complexity of the underlying agent control problem by only reacting to changes in subsequent domain engine frames that the agent perceives (as opposed to reacting to each subsequent frame from the domain engine). That is, the agent builds a model of the domain dynamics from events that the agent perceives, not from frames that the domain engine generates. As a result, the execution of the agent-selected action is executed as long as there are no perceptible changes (by the oracle node of agent f Sims) in the subsequent domain engine frames. For example, in Fig. 23, if the agent starts the execution of the selected domain action (depicted as a green bar) at time t''1, this execution is not interrupted when the domain engine generates frame #2 (because presumably the difference between frames #1 and #2 is not perceptible by the oracle node of agent f Sims). Instead, action execution continues until the domain engine (at time t'3) generates frame #3 (possibly containing some fSims oracle observations that differ from the oracle observations in frame #1). When action execution is suspended after the arrival of frame #k at time t'k, the agent has until time tk+1 to determine new actions to send to the domain engine, which then starts generating frame #k+1. Thus, between times t'k and tk+1, the agent (i) determines its best policy, (ii) executes a focusing policy (a prefix of the best policy) to collect relevant (policy-wise) observations from frame #k, and (iii) starts executing domain actions (applied by the policy). For example, in Figure 23, the agent may generate a new policy at time t'k, which is a new policy that the agent will send to the domain engine.

[0935]

Number

[0936] If the focusing head of f Sim can move substantially faster than the rate at which the domain engine generates new frames, the synchronous focusing agent will succeed in solving the agent / domain engine clock interlock problem. Otherwise, the agent can still be forced to act in a manner that is (by the domain engine) squashed. Note that when the agent is forced to start executing some domain action required by the domain engine (at time t4) for the generation of Frame #4 during the execution of the focusing policy, it is likely to be forced to act early at time t4. To solve this new problem caused by an inappropriate high-speed f Sim focusing head, the CCE also introduces an asynchronous focusing agent.

[0937] The CCE Asynchronous Focusing Agent (see bottom diagram in Figure 23) extends the CCE Synchronous Focusing Agent with a critical feature (for AGI): the agent no longer needs to issue a domain action to the Domain Engine when it is time to start generating another domain frame. That is, the Domain Engine can now also generate frames without any action issued by the agent. This feature has two significant consequences: (i) the agent can now take much longer (than the duration between Domain Engine frames) to search for its best policy and execute a focusing policy action. Only perceptible changes (by Sim oracle nodes) in the Domain Engine frames will interrupt whatever the agent is currently doing; (ii) if the agent does not discover a policy worth pursuing at any given time, the agent need not act, but instead may remain in an idle / daydream state until some newly generated frame that provides a new perceptible observation, considering which a policy worth pursuing may now be discovered. These two consequences are best illustrated in Figure 23. (i) Following the arrival of frame #1 (at time t'1), the agent has plenty of time to explore policies and execute the focusing policy. There is no need to pause at time t2 (when generation of frame #2 begins). Moreover, there is no need to even pause it at time T'2 (when generation of frame #2 ends, unless a change between frame #1 and frame #2 is perceived by the oracle node of fSim). Regarding (ii), consider the situation at time T'3 when domain engine frame #3 arrives. Now, the agent starts exploring policies, but if there is no policy worth pursuing (e.g., Q(a)=0, for all permissible actions), the agent can effectively start idling / daydreaming (or start executing a longer horizon policy rollout or counterfactual option policy).As can be seen in Figure 23, this idling / daydreaming state of the agent continued until time t'4, when, as a result of some perceptible observation from the newly arrived frame #4, the agent finally stumbled upon a course worth pursuing. In summary, the internal clock of the CCE asynchronous focusing agent is completely decoupled from the internal clock of the domain engine. The agent's current (external or internal) activity can only be interrupted by the arrival of a newly generated domain frame, provided that that frame change is actually perceived by the oracle node of the CCE f Sims.

[0938] Implementation details of the CCE execution agent introduced above are provided next.

[0939] 5.1 Synchronous non-focusing agent algorithm. Pseudocode for a synchronous non-focusing agent is provided in Algorithm 12 (see Figure 24L). The algorithm is run for a desired number of frames Fmax by iterating in lines 1-9 over the frames Fcounter=1,2,...,Fmax generated by the domain engine. For the current frame, the algorithm first receives (in line 2) the set of currently allowable domain actions from the domain engine, and then (in line 3) calls the function CalculateActionUtilities (see Algorithm 13, Figure 24M for details) to calculate the action for each

[0940]

number

[0941]

number

[0942] Once the action a* to be performed in the current frame has already been established, the algorithm sends it to the domain engine (line 6) to generate a new frame. The agent then waits for the domain engine to receive the newly generated observation frame and (optionally) the corresponding immediate reward from the domain engine. (The explicit modeling of the arrival of immediate rewards is optional, since for each fNet domain 1 corresponding to some fSim observation k,

[0943]

number

[0944]

number

[0945] Finally, the CalculateActionUtilities() function (Algorithm 13, see Figure 24M) calculates all

[0946]

number

[0947]

number

[0948] 5.2 Synchronous focusing agents. Pseudocode for a synchronous focusing agent is provided in Algorithm 14 (see FIG. 24N). The algorithm calculates the desired maximum number of focusing movements (also known as saccades) per domain frame, S max For the desired number F max Therefore, the domain engine frame counter F counter is initialized to 1 (the agent starts when frame #1 is generated), while a counter S of focusing operations performed in the current frame counter is reset to 0 (lines 1-2). Then, in lines 3-11, the agent counter <F maxAs long as f is , it interacts with its Sims and domain engines by performing permissible focusing and domain actions. Specifically, (lines 4-5) it first determines its set of permissible actions. These are counter max If, then, are the currently permissible domain actions (specified by the domain engine) as well as the currently permissible focus actions (specified by f Sims). The latter also automatically determines (in line 6) the current set of oracles that are the agent's currently perceptible observations (the selected nodes of the f Sim intermediate representation tree, given the current frame). The agent action a* to be executed is determined (lines 7-9) to be the one that maximizes Q(a), given {Q(a)} returned by the call to CalculateActionUtilities() (see above). Alternatively, a* can be a small probability

[0949]

number

[0950] Once the action a* to be performed is established, the agent ENTERS()s the action a* into all fNets (line 10). Then, depending on whether a* is a focusing action or a domain action, the agent acts differently. For a focusing action a* (lines 11-14), the agent increments a counter Scounter of focusing actions performed in the current frame, then executes this focusing action (by touching the fSim corresponding to a*), intercepts the returned observation (the shading of the fSim node pointed to by the fSim focusing head), and then ENTERS()s it into each of the fNets.

[0951] ​In contrast, if the action a* to be executed is a domain action (lines 16-29), the agent first resends its Scounter and increments its frame counter Fcounter (because domain actions eventually return a new frame), and then sends a* to the domain engine to generate a new frame. The agent itself does not have direct access to this new frame. Instead, the agent has access to the f Sim intermediate representation of this frame. Thus, the agent receives from f Sim a list of oracle observations that have been modified due to the presence of the new frame. After calling ENTER() (in line 19) and optionally inputting into each fNet the nonzero discretized immediate reward returned by the domain engine, the agent returns to the list of modified oracle observations. If the list is empty (line 28), the algorithm returns to line (17) to execute a* for yet another domain engine frame. Otherwise,

[0952]

number

[0953] 5.3 Asynchronous Focusing Agent Pseudocode for an asynchronous focusing agent is provided in Algorithm 15 (see FIG. 24O). The algorithm calculates the desired maximum number of focusing movements per domain frame, S max For the desired number F max Thus, the domain engine frame counter Fcounter is initialized to 1 (the agent starts when frame #1 is generated), while the counter Scounter of focusing actions to be performed in the current frame is reset to 0 (lines 2-3). At each of the next Fmax frames, the domain engine

[0954]

number

[0955]

number

[0956] Each loop iteration consists of two distinct parts: In part 1 (lines 5-21), the agent:

[0957]

number

[0958]

number

[0959]

number

[0960]

number

[0961]

number

[0962] Part 2 is devoted to the creation of a new frame by the domain engine. A new frame is created: (i) an agent is executing its domain action

[0963]

number

[0964]

number

[0965]

number

[0966]

number

[0967] This concludes the description of the CCE execution agent. The remainder of Section 5 outlines auxiliary features of the CCE execution agent that can aid in its decision making.

[0968] 5.4 Counterfactual Control. One way to improve the performance of a CCE-performing agent is to retrospectively imagine having performed different actions in the past (e.g., forgetting the most recent execution of a random action that led the agent to a bad outcome). This method starts by tracking the actions the agent performed in its decision stages, as well as the f-Net states at these decision stages. In light of evidence (observations following the execution of these actions), the agent then revisits past decision stages, instantiates the f-Net to its corresponding past states, and recomputes the best policy from each such decision point. The first action of such a computed policy is then input into the f-Net, encoding into the model that this action was the most recently executed action at the decision point. Retrospective execution of alternative (better) actions at such (imagined) policy decision points allows the agent to potentially reconsider executing policies that led to these decision points.

[0969] Another way to improve the performance of a CCE execution agent is to counterfactually consider the execution of high utility options that are currently deemed infeasible. This can be achieved by re-examining the options and finding reasons for their infeasibility. The option unrolls one or more switches of some unit u.

[0970]

number

[0971] 5.5 Multitasking and Daydreaming. Finally, an asynchronous focusing agent (from Section 5.3) can effectively use its idling time towards better policy exploration. This can be achieved, for example, by planning with options (for actions), or by using longer policy rollouts, or by using master-slave level counterfactual plans. Alternatively, the agent can predict the continuation of its attention sequence into the more distant future (involved in intraday reaming) and imagine that it is actually there in the corresponding future state. In such a future state, it can find options that would have been infeasible but could potentially be made feasible from the agent's current state. Thus, the agent can execute a preemptive action policy that would make the future option policy feasible. Since all this occurs while the agent is idling (e.g., waiting for some observation in its currently executed plan), the agent effectively multitasks between the currently executed plan and some preemptive action plan. This remarkable agent ability would undoubtedly manifest itself as a superior agent intelligence in its domain of expertise. 6.0 Hardware Acceleration 6.1 Reserved: Implementation using CPU. 6.2 Reserved: Implementation using GPGPU. 6.3 Reserved: Implementations using Custom ASICs. 6.4 Supplementary information.

[0972] As an alternative to the UNPACK(...) function (Algorithm 9, Fig. 24I),

[0973]

number

[0974]

number

[0975] While various examples of the present disclosure have been shown and described herein, such examples are provided by way of example only. Numerous variations, changes, or substitutions may be made without departing from the present disclosure. It will be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0976] Example 1 - Storage Terms and Definitions Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0977] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Any reference to "or" herein is intended to include "and / or" unless specifically stated otherwise.

[0978] As used herein, the term "about" refers in some cases to an amount that is approximately the same as the amount recited. As used herein, the term "about" refers to an amount that is 10%, 5%, or 1% closer to the amount recited, including increments thereof. As used herein, the term "about" in reference to a percentage refers to an amount that is 10%, 5%, or 1% greater or less than the recited percentage, including increments thereof.

[0979] As used herein, "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together.

[0980] Although preferred embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. The present disclosure is not intended to be limited by the specific examples provided herein. The present disclosure has been described with reference to the foregoing specification, and the description and illustration of the embodiments herein are not meant to be interpreted in a limiting sense. Numerous variations, changes, and substitutions may be made without departing from the present disclosure. Furthermore, it should be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions described herein, which depend on various conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in implementing the present disclosure. Therefore, it is contemplated that the present disclosure shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the present disclosure, and that systems, methods, and structures within the scope of these claims and their equivalents are covered thereby.

Claims

1. 1. A system for determining domain observations of an environment, comprising: a domain engine module configured to generate a first set of domain observations from the environment; an active sensing module configured to: (i) receive the first set of domain observations; (ii) generate a second set of domain observations from the first set of domain observations; and (iii) generate a first set of sim actions; a fractal network module configured to (i) receive a second set of the domain observations, (ii) receive the first set of sim actions, (iii) generate one or more fractal networks associated with the second set of domain observations and / or the first set of sim actions, (iv) generate an observation sequence from the one or more fractal networks, and (v) compare the observation sequence with the first set of domain observations; an execution agent module configured to (i) generate a second set of sim actions based on the sequence of observations, and (ii) send the second set of sim actions to the active sensing module. The second set of sim actions causes the active sensing module and / or the domain engine to generate another set of domain observations and / or another set of sim actions.

2. The system of claim 1 , wherein the domain engine module generates the first set of domain observations at one or more rates or upon one or more events.

3. The system of claim 1 , wherein the first set of domain observations includes one or more of a temporal observation, a spatial observation, or a spatio-temporal observation.

4. The system of claim 1 , wherein the first set of domain observations includes one or more of a visual observation, a textual observation, an auditory observation, a vector observation, or a scalar observation.

5. The system of claim 1 , wherein the second set of domain observations comprises a transformation of the first set of domain observations.

6. The active sensing module includes one or more active sensing modules, each active sensing module comprising: a focusing head module configured to focus on or select a first portion of each domain view; a data architecture module configured to generate a node associated with the first portion of each domain observation; a perception module configured to generate a perceived observation associated with a second portion of each domain observation, the second portion being proximate to the first portion of each domain observation; The system of claim 1 , comprising:

7. The system of claim 6 , wherein the number of the one or more active sensing modules is dynamically changed in real-time based on receiving at least new types of domain observations.

8. The system of claim 1 , wherein the fractal network module includes one or more fractal networks, each fractal network comprising a computation unit configured to store one or more placeholder values ​​associated with the operation of the respective fractal network.

9. The system of claim 1 , wherein the number of the one or more fractal networks changes dynamically in real time based on receiving at least new types of domain observations.

10. The system of claim 1 , wherein each of the one or more fractal networks is evolved independently of another fractal network to optimize capture of the observation sequence.

11. 2. The system of claim 1, wherein each of the one or more fractal networks iteratively learns using the first set of domain observations, the second set of domain observations, the first set of sim actions, or the second set of sim actions.

12. The system of claim 1 , wherein each of the one or more fractal networks is iteratively trained to reach a confidence level associated with determining a relationship between each domain observation of the first set of domain observations and the observation sequence.

13. The system of claim 1 , wherein each of the one or more fractal networks is configured as an elastic architecture having one or more regions arranged in a linear or non-linear manner.

14. The system of claim 13 , wherein each of the one or more fractal networks is configured to change a location of the one or more regions in the elastic architecture to another location of the one or more domains in the elastic architecture.

15. The system of claim 14 , wherein computational units of one domain are dynamically connected to computational units of other domains during operation of the fractal network module using one or more horizontal electrical connections or one or more vertical electrical connections.

16. The system of claim 15 , wherein each computation unit of the one or more regions is configured with one or more placeholders associated with an arithmetic or logical operation.

17. The system of claim 1 , wherein the execution agent comprises a synchronous non-focusing agent.

18. The system of claim 1 , wherein the execution agent comprises a synchronous focusing agent.

19. The system of claim 1 , wherein the execution agent comprises an asynchronous focusing agent.

20. The system of claim 1 , wherein an executive agent utilizes counterfactual control to instruct the fractal network module to determine future sim actions or future observations.

21. The system of claim 1 , wherein the environment comprises a computational domain or a cyber-physical domain.

22. The system of claim 5 , wherein the transformation comprises a reduced or increased resolution of the first set of domain observations.

23. The system of claim 5 , wherein the second set of domain observations occurred contemporaneously with the first set of domain observations.

24. The system of claim 6 , wherein the data architecture module comprises a quad-tree structure, an oct-tree structure, a kd-tree structure, or any combination thereof.

Images