The memories of the materialized agent

Abstract

A computational structure provides an embodied agent with memory that can be populated and / or authored in real time from experience. The embodied agent (which may be a virtual object, digital entity, or robot) is provided with one or more experiential memory stores that influence or direct the behavior of the embodied agent. The experiential memory stores may include convergence-divergence zones (CDZs), which simulate the ability of human memory to represent external reality in the form of mental images or simulations that can be re-experienced during recall. The memory databases are generated in a simple, authorable manner, allowing for learning or authoring of experiences during the embodied agent's live operation. Eligibility-based learning determines which aspects from a stream of multimodal information are stored in the experiential memory stores.

Description

[Technical Field]

[0001] The embodiments described herein relate to the field of artificial intelligence, and to systems and methods for implementing and using Memory in Embodied Agents. More specifically, but not limited to, the embodiments described herein relate to unsupervised learning. [Overview of the Initiative] [Problems that the invention aims to solve]

[0002] The goal of artificial intelligence (AI) is to build computer systems with human-like capabilities, including human-like learning and memory. Most modern machine learning techniques rely on "offline" learning, where the AI ​​system is provided with cleaned data that has been prepared to learn and limit itself to a specific domain. A significant challenge in prior art remains creating AI systems that experience the world's objects and events in a human-like manner and learn from embodied interactions. Through sensorimotor feedback loops with their embodiments and their environment, such AI agents can influence and guide their own learning. Such agents will understand the meaning of multimodal data streams from the world and retain information in a meaningful and useful way. A further significant challenge is to create embodied agents of flexible AI that can learn from their own experiences and whose memories can be authored or modified by external sources (such as human users). Hierarchical Temporal Memory (HTM) is an approach to replicating human memory based on a computational structure with multiple registers as analogues of cortical layers. HTM is configured to replicate patches of the cerebral cortex. Nevertheless, HTM cannot provide Memory in Embodied Agents, which would enable the embodying agent to learn and develop in real time from sensorimotor experiences. [Brief explanation of the drawing]

[0003] [Figure 1] Figure 1 is a schematic diagram of the CDZ architecture. [Figure 2] Figure 2 is ASOM. [Figure 3] Figure 3 shows the eligibility signals for different modalities. [Figure 4] Figure 4 shows how the eligibility trace creates the eligibility window. [Figure 5] Figure 5 shows the phases of a learning event. [Figure 6] Figure 6 shows a user interface for setting eligibility for different modalities. [Figure 7] Figure 7 shows the ASOM training display. [Figure 8] Figure 8 shows the query viewer input fields. [Figure 9] Figure 9 shows the user interface for specifying query patterns. [Figure 10] Figure 10 shows the LTM and STM displays. [Figure 11] Figure 11 shows a working memory system (WM system). [Modes for carrying out the invention]

[0004] The computational structure provides an embodied agent having memory that can be input and / or authored in real time from experience. The embodied agent (which may be a virtual object, digital entity, or robot) is provided with one or more experience memory stores that influence or direct the behavior of the embodied agent. The experience memory stores may include a Convergence-Divergence Zone (CDZ), which simulates the human memory's ability to represent external reality in the form of mental images or simulations that can be re-experienced during retrieval. The memory database is generated in a simple and authorable manner, allowing experience to be learned or authored during the live operation of the embodied agent. Eligibility-based learning determines which aspects of a multimodal stream of information are stored in the experience memory store. Experiential memory store

[0005] In one embodiment, experiences experienced by an agent are stored in one or more experience memory stores. “Experience” should be broadly interpreted as anything that an embodied agent can sense or perceive, such as objects, events, emotions, observations, actions, or any combination thereof. The experience memory store(s) can store dimensionally reduced representations of experiences in neural network weights. Convergence-Divergence Zone (CDZ)

[0006] In one embodiment, the experiential memory store is implemented as a convergent-divergent zone (CDZ). The CDZ is a network that receives convergent projections from sites where activity is recorded and returns divergent projections to the same sites. Patterns in the CDZ hold “temperaments” that complete or act in accordance with partially presented perceptual patterns. Hierarchical upstream associated memories associate combinations of activities in lower-level sensory and / or motor maps to form implicit memories (e.g., aggregated properties of objects) that enable downstream reconstruction of component properties. For example, an experiential memory store that stores experiences of objects that can be used for object classification can be implemented using CDZs as follows: Each unimodal object classification path is a hierarchy of CDZs, and explicit maps of objects are constructed during perception and reconstructed during retrieval. By activating a pattern in any single lower-level modality, patterns in higher-level multimodal CDZs can be triggered if one is learned. This activity can then trigger activities that flow “top-down” into other CDZs, activating patterns learned by the experiential memory store that were associated with the initial pattern.

[0007] Figure 1 shows the diagram of CDZ1. The multimodal CDZ sits above each higher-level unimodal CDZ. The association between two modalities X and Y is not held by a direct link from X to Y, but in a separate region ("converging zone") Z that is independently linked to both X and Y. Representations converge from multiple regions into the region of region Z. The declarative representation of the convergeding zone remembers the associations between stimuli within the subregions. To reveal the associations, patterns are explicitly activated. When a convergeding zone representation is activated, it reveals the activity of associated patterns within the set of subregions and thus acts as a "diverging zone" that diffuses activity from a single region to a range of regions.

[0008] The convergence-divergence zone can be implemented using a map that can receive input from several modalities and then be activated by any of the modalities. The map may be an Associative Self-Organizing Map (ASOM) that associates different inputs by taking activation maps from lower-level maps and associating concurrent activations. The ASOM receives an input vector having a size and number of input fields corresponding to neuron weight vectors, where each input field represents a different modality or input type. When trained on many inputs, the map learns topological groupings of similar inputs. The ASOM can function in both a "bottom-up" construction mode and a "top-down" reconstruction mode. The ASOM can generate predictions that can be compared against input information. The lowest-level (unassociated) sensory, motor, or other activity sites can be implemented as maps such as self-organizing maps, or in any other suitable way.

[0009] Figure 3 shows the mapping between lower-level SOMs and upper-level ASOMs, where the ASOM is the structurally constructed input field of the convergence zone (CDZ). Lower-level SOMs include sensorimotor inputs corresponding to visual, audio, touch, neurochemical (NC), and positional modalities. In the hierarchically structured set of CDZs, lower-level CDZ SOMs provide inputs to upper-level CDZ SOMs. Upper-level ASOMs, functioning as convergence-divergence zones, include Visual-Audio-Touch (VAT), Visual-Motor (VM), and Visual-Neurochemical (VNC). The Visual-Audio-Touch (VAT) ASOM convergence-divergence zone is associated with the positional modality within the upper-level VAT-Activity-Position ASOM.

[0010] CDZ enables real-time learning systems to store multimodal and emotional memories. This is how, for example, when an embodied agent "imagines" a dog or hears the word "dog," one or more neurons in the higher-level ASOM representing the dog are activated. The higher-level ASOM has pointers to lower-level sensory maps for vision (showing an image of a dog), audio (hearing a dog barking), and even to emotional state maps that recreate the emotions the embodied agent felt when they first encountered a dog. Modality

[0011] "Modality" should be broadly interpreted as a mode of something existing, including its representation, manifestation, or experience. Objects and / or events can be experienced in different modalities, including but not limited to visual, audio, touch, motor, and neurochemistry. In one embodiment, each modality input is represented and / or learned by an individual SOM. An architecture may be used that includes maps associated with each modality, so that when two or more modalities are experienced simultaneously, the combination is stored in a higher-level (combined) map as a pointer to each of the two senses in their original sub-maps. The combined map can be activated by inputs corresponding to any of the modalities it associates. If input is received from only one of the modalities, the corresponding representation from the other modality can be predicted.

[0012] Visual input can be streamed to the embodied agent in any preferred manner. In one embodiment, vision is provided to the embodied agent via a camera capturing a real-world environment. Vision can be delivered from a screencast of a user interface or from a computer system in other ways. Thus, the vision of the embodied agent can be the real world, which may include seeing a human user via the camera, or a “virtual world” or a computational representation (such as a screen representation or a VR / AR system representation), or any combination of the two. Both the “real world” and the “interface” fields of view can be represented to the embodied agent so that the embodied agent has two distinct fields of view. Each field of view may have an associated splendor map that controls attention. In one embodiment, only a single splendor region across these two maps has been selected for attention so far, and so the two fields of view can be treated as a single field of view having two parts when it comes to attention routines. Subregions of camera input can be automatically mapped to smaller regions of video input corresponding to a virtual “fovea,” i.e., where the embodied agent’s eyes are directed. The foveal image subregion can be further processed by modules, such as an influence classifier and / or an object classifier. This allows the embodied agent to focus on only a small portion of the camera input, thereby reducing dimensionality. In one embodiment, a 28x28 RGB foveal image is provided. Peripheral images can also be processed, but at a much lower resolution.

[0013] Audio input can be delivered via a microphone that captures the waveform to be processed by the auditory system. In one embodiment, the acoustic characteristics are analyzed using FFT and other techniques to create a spectrogram, which is used as input to the auditory SOM (e.g., a 20 × 14 (f × t) spectrogram). The auditory SOM learns a frequency-specificity map of the audio input. Alternatively and / or additionally, digital audio inputs, such as those from audio files or streaming from a computer system, can be delivered to the embodied agent. The acoustic signal can be analyzed via a deep neural network that provides a vector of values ​​corresponding to input words. These are fed to a second independent auditory SOM that learns word mappings. The frequency-specificity and word maps can be further integrated by a higher-level auditory ASOM, which is the final representation of the audio modality.

[0014] The tactile sensation can be provided to the embodied agent based on its interaction with the virtual environment. For example, whenever a part of the body of the embodied agent intersects with another object within the environment of the embodied agent, the object intersection can trigger the tactile sensation of the embodied agent. Such a tactile sensation can be associated with the proprioceptive map of the body of the embodied agent, the map of the environment of the embodied agent, and / or any other modality. When the embodied agent touches a specific "touchable" object in the virtual world, a collision is detected, and activity is triggered in the mechanoreceptors on the effector (e.g., finger) of the embodied agent. The tactile sensation can be provided to the embodied agent via a computer input device such as a mouse, keyboard, or touch screen. For example, by "touching" the screen, a part of the body of the embodied agent protrudes, and the finger or mouse cursor (on the touch screen) is "brought into contact" with the mechanoreceptor map. Symbolic input (e.g., keyboard input) can be mapped to any tactile sensation, for example, to the object texture. For example, the tactile object type SOM can map different object textures. The shape of the object can also be registered via a tactile system involving both touch and motor movement.

[0015] The "position" modality can represent the foveal position including the x and y coordinates of the fovea of the embodied agent. The coordinates can be directly converted to a 10×10 activation map via the SOM from position to activity.

[0016] Interoceptive sensations are the embodied agent's perceptual sensations of the internal state of the embodied agent's body. The interoceptive state space map is formed by taking inputs from signals representing the momentary state of the body, such as hunger, thirst, fatigue, heart rate, pain, and aversion. Neurochemical parameters represent physiological internal state variables that are part of the emotional system. The interoceptive map represents the state space of the embodied agent. Examples of neuromodulators that can be modeled include acetylcholine for motor function, cortisol as a stress indicator, and oxytocin for social bonding. The basic expressions of primary emotions can be mapped into a high-dimensional neurochemical space, which provides a mapping from continuously felt intuitive states to discrete psychological categories by modulating behavioral responses. Interoceptive sensations can contribute to the decision-making of the embodied agent because events are associated with the emotional neurochemical state of the body, and as a result, the evoked emotions of imagined events are factors in decision-making.

[0017] The proprioceptive system provides an embodied agent with proprioceptive awareness through proprioceptors regarding the constitution of the embodied agent's body, including the position of the agent's effectors (e.g., hands, feet, head, and the constitution of the agent's torso). The proprioceptive map can include information regarding the angles of each joint delivered from the skeletal model of the embodied agent's body. In a more detailed biomechanical model of the embodied agent's muscle tissue, the proprioceptive map can also include information regarding muscle stretch and tension. Motor modalities can be used to map types of actions.

[0018] Individual words can be associated with the representation of an object, action, event, or concept through written words, auditory phoneme representations, and / or other symbols. One or more symbols associated with the representation of a concept can be remembered as modalities associated with the sensory modality representing the concept.

[0019] Any other suitable modalities (or hypothetical representations thereof) such as taste and smell can be implemented. Specific aspects of a modality can be modeled as modalities themselves. For example, the visual modality can be divided into several modalities, including light modality, color modality, and form modality. Internal sensations such as temperature, pain, hunger, or balance can be modeled. Directly author the experiential memory store. It is possible to store the trained neural network (such as a SOM) with these post-training weights in an embodied agent that has not directly experienced the weights. In this way, the "blank" embodied agent can be provided with knowledge (e.g., of objects) embedded in the neural network weights of its experiential memory store(s). Memory database (memory file)

[0020] In one embodiment, representations of experiences can be stored in a memory database in addition to the experience memory store. The memory database can be automatically entered and / or authored through the experiences of the embodied agent. A user or automated system can retrieve memories stored in the memory database, author new memories in the memory database, and / or delete memories. Raw data corresponding to each experienced representation of a modality can be stored in the memory database and associated with the corresponding experience. For example, components of a memory relating to a visual modality can be linked to image files (e.g., JPEG, PNG, etc.), and components relating to an auditory modality can be linked to audio files (e.g., MP3).

[0021] The memory database can be implemented in any preferred form, for example, as a database and / or folder that stores a collection of files. In one embodiment, the memory database is a CSV file that stores experiences. A CSV entry may contain or point to a representation of the raw data associated with the experience corresponding to the entry. By storing the experiences as associated images or other raw data corresponding to raw inputs, the experiences can be replayed / processed by an agent. The embodied agent can learn their inputs as if the embodied agent were experiencing them.

[0022] In one embodiment, during the live operation of the embodied agent, the embodied agent simultaneously stores memories of experiences in both an experience memory store and a memory database. For example, the experience of a barking dog can be stored as a multimodal memory in the experience memory store, and can also be stored as attributes of an entry corresponding to the experience in the memory database, including other related multimodal data such as images, sounds, emotional valence, and text / voice utterances.

[0023] Storing experiences in a file may also involve storing metadata or additional data about the experience, such as the time the event occurred (timestamp), the GPS location of the event, or any other contextual information about the experience. Memory is entered through experience.

[0024] In one embodiment, memories stored in a memory database are input from the real-time experiences of the embodied agent during the live operation of the embodied agent. The agent interacts with sensory streams from the real and / or virtual worlds, as described in New Zealand Provisional Patent Application No. NZ744410, titled "Machine Interaction," which is also assigned to the assignee of the present invention and incorporated herein by reference.

[0025] As described herein, an embodied agent can selectively learn new, emotional, or user-signaled experiences through experience. In an embodied agent whose experience memory store is implemented as a CDZ, memories are stored in the CDZ. Whenever a new memory of an experience is stored in the CDZ, a representation from the lower-level SOM is saved as an attribute and / or file of the new entry in the memory database. Train the experiential memory store via a memory database.

[0026] The memory database can be used to train the experiential memory store. Entries in the memory database are provided as training input to the experiential memory store during integration. The memories encoded in the experiential memory store enable the agent to recognize and predict objects, concepts, and events. As an example, a user can generate a set of input files for a specific learning domain. For instance, an agent can become a "dog expert" without experiencing dogs in live operation by being provided with a memory database containing symbols including dog names, spectrograms of their barking sounds, and images of different dog breeds with associated modalities such as the emotional responses the dogs would evoke.

[0027] In CDZ implementations, entries in the memory database are used to retrain the CDZ, modifying the weights of the underlying convergence / divergence zones (e.g., SOM / ASOM). During training, the raw files / data corresponding to the entries are reread by the experience memory store, one experience at a time. Taking object learning events as an example, raw data corresponding to visual, auditory, and touch modalities are loaded to trigger learning events. Long-term memory learning events occurring during memory integration can occur on a much faster timescale than real-time learning, as discussed in the section titled "Memory Integration." In one embodiment, the raw files used to "train" the agent can be displayed to simulate the agent's "daydreaming" as the agent "relives" or "reimagines" past experiences. Reconstructing memories

[0028] Entries in a memory database can be reread to reconstruct memories. For example, they can be used to train short-term memory experience memory stores to generate "virtual events," or to train long-term memory experience memory stores during memory integration. It may also be possible to reconstruct the raw sensor inputs (such as images) that triggered the learning events from the experience memory stores as raw sensory inputs are stored in the weights of neurons in lower-level maps. However, potentially, the resulting weights in a neural network may be a blend of several input instances, since several different input vectors can modify the weights of a single neuron. Because a memory database explicitly stores individual input vectors and their constituent input fields as separate entries with associated attributes, it provides a way to precisely reconstruct individual experiences. Modify or delete memories

[0029] Memories can be selectively modified by the user, for example, by modifying entries in the memory database (explicit modifications such as changing the valence of an object) or by deleting an entire entry. By deleting all entries, the entire memory of the embodied agent can be deleted, leaving a blank state. In one embodiment, at each integration, the experiential memory store is cleared and completely re-entered by training with the updated memory database (which may include edited or deleted entries). In an experiential memory store that is a SOM, clearing the experiential memory store can be achieved by randomizing all neuron weights.

[0030] In other embodiments, instead of clearing the entire experience memory store, updated or modified experiences can be placed in the experience memory store and selectively deleted by “discarding” certain data points to form the experience memory store. In the “forgetting” model, experiences may be timestamped or otherwise marked to indicate the freshness of the memory, and older events may be “forgotten” by deleting them from the experience memory store and / or memory database. Authoring memories Instead of requiring the agent to have new experiences and create new memories, memory entries corresponding to experiences can be directly "embedded" into the agent's memory. This creates an embodied agent that is directive, artificially manipulated, and tangible. For example, the agent can be programmed to have directed autonomous responses to experiences (such as negative reactions to specific stimuli). Thus, entries in the memory database can be "authored" by external tools, as well as learned directly from the real-time sensorimotor experiences of the embodied agent. Authoring using a text corpus

[0031] Memory authoring can be done in a context with a text corpus. An example of a marked-up text corpus for authoring an event memory is as follows: [timestamp] A red car (image, sound) drove to the left (location) (action). I didn't like it (emotion).

[0032] Real-time sensorimotor context can reflect the word choices (like, dislike) and deictic and emotional states of the embodied agent. This can be achieved by providing a lookup table of raw inputs (images / sounds / feelings, etc.) associated with symbols such as words. This allows for the rapid creation of inputs for learning events through sentences. Data matching corresponding words in the lookup table is retrieved to train the experiential memory store and / or to create detailed entries associated with the raw data in the memory database. Embedded agents with existing knowledge of objects, actions, and emotions can author events by associating the components of the event using syntactic structures.

[0033] Memories can be categorized, labeled, or tagged in a manner that facilitates the locating, modifying, and / or deleting of individual memories. A user interface can be provided to facilitate the user viewing and editing the memories of the embodied agent. Implementation using a Self-Organized Map (SOM) Self-organized map

[0034] Both modalities and convergence / divergence zones can be represented using self-organized maps (SOMs), also known as Kohonen maps, which are unsupervised learning-based memory structures. To provide a discretized / quantized representation of this data, an SOM (which may be 1, 2, 3..., or n-dimensional) is trained on the dataset. This discretization / quantization can then be used to classify new data within the context of the original dataset. Weighted distance function

[0035] In conventional SOMs, the difference between the input vector and the neuron's weight vector is calculated using a simple distance function (e.g., Euclidean distance or cosine analogue) across the entire input vector. However, in some applications, it may be desirable to weight certain parts of the input vector (corresponding to different input fields) more highly than others.

[0036] In one embodiment, a Joint Self-Organized Map (ASOM) is provided for multimodal memory, where each input field corresponding to a subset of input vectors contributes to a weighted distance function by a term called the ASOM alpha weight. The ASOM computes the difference between a set of input fields and the neuron's weight vector not as a monolithic Euclidean distance, but by first splitting the input vector into input fields (which may correspond to different attributes recorded in the input vector). The differences in the vector components within different input fields contribute to the total distance by different ASOM alpha weights. The activity obtained as a single result of the ASOM is computed based on the weighted distance function, and different parts of the input vector can have different semantics and their own ASOM alpha weight values. Thus, the overall input to the ASOM also encompasses any inputs that may be associated with it, such as different modalities, activities from other SOMs, or something else. Figure 2 shows the ASOM architecture for integrating inputs from several modalities. Inputs to ASOM

number

number

[0037] The ASOM 3 in Figure 2 consists of N neurons and has weight vectors corresponding to all inputs.

number

number

number

number

[0038] In the formula, α k This is the bottom-up mixing coefficient / gain (ASOM alpha weight) of the k-th input field. k is a distance function specific to the input field. Any suitable distance function(s) may be used, including, but not limited to, Euclidean distance, KL divergence, and cosine-based distance.

[0039] In one embodiment, the weighted distance function is based on the Euclidean distance as follows.

Number

[0040] Where K is the number of input fields, and α i is the corresponding ASOM alpha weight of each input field, D i is the dimensionality of the i-th input field, and x j (i) or w j (i) is the j-th component of the i-th input field or the corresponding neuron weight, respectively.

[0041] In some embodiments, the ASOM alpha weights may be normalized. For example, when the Euclidean distance function is used, the activity ASOM alpha weights are typically made to sum to 1. However, in other embodiments, the ASOM alpha weights are not normalized. Not normalizing can result in a more stable distance function (e.g., Euclidean distance) in certain applications such as ASOMs with a large number of input fields or high-dimensional ASOM alpha weight vectors that vary dynamically from coarse to dense. Method of Sampling Memory: Dreaming and IOR

[0042] It may sometimes be desirable to randomly reconstruct the items stored in the SOM. This occurs, for example, in the construction of pseudo-training items during the integration of long-term memory or the random generation of motor movements during motor babbling. In these cases, the training record of the SOM drives the probabilistic selection of the SOM neurons to be reconstructed. Sampling can be combined with an inhibition-of-return (IOR) process to sample from the complete set of trained values. Training Record When reconstructing from the complete activity of the SOM, every neuron contributes proportionally to the similarity of its weight vector to the input vector, regardless of whether these neurons are trained to represent meaningful hypotheses or whether they contain initial random noise. To provide a cleaner, reconstructed output, more weights may be given to more trained neurons (ignoring untrained neurons). The adaptation amount for each neuron can be recorded as a value between 0 and 1, accessible in the SOM parameter "training record". The training record is the additional scalar weight of each neuron, initialized to 0 and connected to a fixed input of 1. Thus, each time this particular neuron is trained, the neuron's training record increases proportionally to the current (potentially adapted, due to good matching) learning rate, because it is either a winner or near a winner. This means that during the training process, the training record will rise towards 1. The average of the training records of all neurons in the map ("map occupancy") indicates the free capacity of the map for learning new inputs without overwriting old ones. A "maximum occupancy" of 1 indicates a "full / crowded map" (no free capacity), while a "maximum occupancy" of 0 means an untrained map. The training record is used in the activity calculation of SOM, where the value of the activation mask (item m) is used. i It can function as a ). In the Bayesian term, uniform m i Instead of using a flat prior probability (where all hypotheses are equally likely), this is equivalent to adopting prior probabilities based on the frequency of observations. That is, the resulting probability distribution is conditional on the assumption that the input is one of the previously observed inputs from which the SOM was trained. Training records can decay over time, meaning that while exploration may avoid areas of recent training, they can be reused for new memories if the areas haven't been reactivated for a long time. How training records store training history is adjustable via the training record decay parameter. A training record decay value of 1 means no decay. A training record decay value less than 1 means the training record reflects only the most recent training (with novelty determined by a value between 0 and 1). Inspect content by restructuring weights in a top-down manner.

[0043] In a connected SOM hierarchy where the activity of lower-level SOMs provides input to higher-level SOMs, the activation flow can be reversed during top-down reconstruction. The reconstructed input from the higher-level SOM provides a top-down signal to the lower SOM: the expected activation pattern. This signal can be fed into the top-down bias field of the lower SOM, which can be combined with the activation pattern that the lower SOM derives from its own input.

[0044] The raw content of "memories" stored in neurons can be retrieved, and depending on the training conditions and SOM parameters, memories can be equal to individual events or a blend of several events (for example, small sigma and large learning rate = "sharp" individual memories, larger sigma and smaller learning rate result in generalized and blended memories). Configuring ASOM for high-speed learning

[0045] Backpropagation-based learning methods require slow learning via small weight updates, but the localist nature of SOM neuron representations allows them to learn input patterns very rapidly, even with a single exposure. The problem encountered by learning inputs "fast" (after only a few presentations) in conventional SOMs is the overwriting of previously encoded inputs. Distinct training items (or training items that are considered distinct for the purposes of, e.g., members of different classes) should be isolated within the SOM by encoding them into separate neurons or regions. At the same time, items that are sufficiently similar to each other should be encoded into the same neuron or region. Unlike previous attempts to associate using slowly learning SOMs, the ASOM described herein can learn "fast". The ASOM can be configured to learn fast by selecting high learning constant / learning frequency values ​​so that a given input can be encoded by a single SOM neuron (or region) with a single exposure. However, varying the learning constant is not sufficient to enable practically fast learning of large sets of items.

[0046] It is possible to determine whether an input is "new," and if the match is not close enough, the "winning neuron" is not overwritten; instead, a different neuron is selected. A "best match threshold" parameter can be defined to control whether an item presented to the ASOM is considered "new" or "old." The "best match threshold" is a threshold for the (raw-denormalized) activity value of the SOM neuron that responds most strongly to the input item. If this value is below the "best match threshold," the item is considered "new"; otherwise, the item is considered "old." New items are stored in the SOM as separate patterns, and old items update existing patterns.

[0047] When a new item is encountered, the "search method" parameter determines which neuron to assign to encode the new input. Any suitable search method can be used. Examples include: • Noise for input search: Add random noise to the current input and find a new winner based on a Gaussian activation function applied to the distance to this modified input. • Noise for Activation Search: New winners are selected from a composite activation map, which is a mixture of the original activation map and a quadratic map filled with random noise. The mixing coefficient of the quadratic map is called comparison noise and determines how distorted the original map is. A small value of comparison noise triggers a local search of the original winner's neighborhood.

[0048] Instead of mixing activity with noise, the secondary map can be set to anything that encodes a bias toward or away from a particular region of the SOM, for example, a value that inversely reflects how frequently and how active each neuron has been recently, in order to ensure that previously winning neurons are avoided and to facilitate more uniform input to the SOM. Particularly useful methods include tracking the amount of training each neuron has received (in total or recently), the so-called training record, and filtering out winner selections from trained regions (involving previously untrained / dead neurons).

[0049] The recording of each neuron / region in the map, the degree of training it has received, and the competition for winning neurons still depend on similarity to the input, but are biased away from regions that have received more training. The network is input uniformly, and "dead neurons" (never trained due to poor initial weights) are reduced. By using the reciprocal of the training records as activation noise, we ensure that any unused neurons are assigned first.

[0050] To preserve the topographical organization of the SOM and place new winners in the neighborhood of the original winners, the "comparison noise" can be set to a small value. When the comparison noise is small, the original activation still has a strong influence, and therefore new winners are more likely to come from the neighborhood of the old winners. It is then trained with the current input, and the original winner encodes what it previously had, and the new input is represented by nearby neurons rather than overwriting it.

[0051] Setting a pattern on a map of values ​​that inversely reflects how frequently and how active each neuron has been recently (thus promoting more uniform input by the SOM by engaging previously unused neurons and ensuring that no first winner is selected) can be computed using pseudocode for the SOM isomorphism vector as follows: [Table 1]

[0052] When an item is considered "old," it is stored in the SOM within a region where some learning has already occurred. In a standard SOM, if the same item is presented repeatedly, the region representing this neuron will develop, potentially growing in size and eventually dominating the entire SOM. This is an inefficient use of the SOM. To control this effect, the "best match learning multiplier" parameter adjusts the learning frequency of the SOM according to the activity of the winning neuron. If the "best match learning multiplier" is set to zero, exactly repeated items will not induce any new learning within the SOM. If it is set to 1, there is no adjustment of the learning frequency of the original SOM. The multiplier M for learning frequency can be calculated using a formula such as: M = 1 - raw winning activity * (1 - best match learning multiplier). A non-zero value with a low best-match learning multiplier may be desirable, rather than zero, because more neurons in the neighborhood of a perfect match can fit towards that value, and the reconstructed "soft" output also reflects how often different values ​​were encountered, meaning that even in the case of a perfect match, some training is good.

[0053] As mentioned earlier, a problem encountered with fast-learning SOMs is that a high learning frequency increases the risk of overwriting neurons. At a low learning frequency, weights are averaged out, but not completely overwritten. Depending on the values ​​of its parameters, an SOM can be configured for slow or fast learning. Slow learning (as described by a standard Kohonen SOM and analogous to cortical learning in the brain, allowing for the generalization / blending of individual memories) is characterized by a smaller learning frequency, a higher value for the neighbor size sigma, and novelty detection disabled by setting best_match_threshold=0, for example. Fast learning is characterized by a maximum learning frequency, a very small sigma, and a high best_match_threshold, analogous to hippocampal learning in the brain, and can behave like a probabilistic lookup table (having a high degree of representing individual experiences separately / orthogonally and accurately). Because the range of the above parameters is continuous, a mixture of fast and slow learning is achievable with SOMs. It is possible to adaptively reduce the learning frequency when the SOM is crowded (in terms of its map occupancy, as described in the training record section). Next, the SOM automatically switches to a standard slow-learning SOM (because continuing to learn quickly in a complete map would mean overwriting / forgetting old knowledge). Reducing the learning frequency allows new memories to blend with the oldest, most similar ones. In one embodiment, the "speed" of learning depends on the SOM capacity. With a sufficiently large SOM, it can be configured to learn individual memories quickly (even in one shot) and with high accuracy. The transition to more gradual learning may occur as the SOM approaches its full capacity (so that old memories are blended with new ones rather than completely replaced). To monitor how much capacity (i.e., unused neurons that can be trained without overwriting old memories) remains, map occupancy can be defined as the average number of training records per neuron, i.e., total_i(training records[i]) / map_size. A value of 0 means an empty / untrained map, and a value of 1 means a complete map.To transition from a fast map type to a slow map type, the parameter learning frequency, sigma, and best match threshold can be gradually adapted as the map occupancy increases. Alternatively, a separate switch can be made when the map occupancy exceeds a certain threshold, e.g., 90% (0.9). Targeted forgetting

[0054] "Forgetting" everything learned by SOM is achievable by replacing all neuron weight vectors with random noise (in the same way SOM is initialized). However, there are situations where "targeted forgetting" is useful. For example, • To "undo" a recently learned experience that was learned incorrectly. • Forgetting all memories of a specific type, namely all images associated with the sound of gunfire. - Forgetting rare memories (assuming they occurred by chance and were of lower quality than frequently encountered experiences). • Forgetting very old memories (assuming that training was unstable at the start and that the resulting expressions were of low quality).

[0055] Targeted forgetting is controlled by a mask called a "reset mask" (similar to an activation mask). The reset mask is isomorphic to the SOM (i.e., each SOM neuron has one mask value). When the neuron weight vectors are replaced with noise, only the neurons with reset mask = 1 are reset, while the others (with reset mask = 0) are retained.

[0056] Alternatively, the reset mask value may be between 0 and 1, in which case the original weight vector is mixed with random noise using a mixing coefficient determined by the reset mask value. New_weight[i] = (1 - reset_mask[i]) * original_weight[i] + reset_mask[i] * noise

[0057] During a reset, the training record can be updated so that the training record of the reset neuron is cleared (i.e., in the case of a discrete reset mask, the training record for a neuron with reset mask = 1 is 0). In the case of continuous mixing (blurring memory), New_Training_Record[i]) = (1 - Reset_Mask[i]) * Original_Training_Record[i])

[0058] An appropriate reset mask can be set according to the following requirements:

[0059] The reset mask is set to the SOM's most recent activation map (the overall activity of the SOM immediately after training for the experience to be reversed). This causes partial forgetting—blurring proportional to the size of the activity. Alternatively, a discrete reset mask may be created, for example, for a stochastic SOM, by setting the reset mask to 1 for all neurons whose activation is greater than the reset threshold, and to 0 for the rest. Or, for a non-stochastic SOM, the mask value is set to 1 for the winning neurons and to 0 for all other neurons.

[0060] A stimulus is input that causes the associated memory to be forgotten. In the example above, a gunshot is provided on the audio input field, and the ASOM alpha weight of the video is set to 0 (to extract all videos associated with the gunshot). The resulting activation map can be used directly as a reset mask. Alternatively, a discrete reset mask may be created, for example, by setting the reset mask to 1 for all neurons whose activation is greater than a set threshold, and to 0 for the rest. Or, set to 1 for the winning neuron and to 0 for all other neurons. The reset mask is set to 1 - training record, or to its discrete version (if training record[i] < threshold, reset mask[i] = 1, otherwise 0). Training record decay is set to a value < 1 during training. This causes the training record to wear down to zero over time for neurons that have not been "refreshed" by new training. Next, the reset mask is set to 1 - training record, or its discretized version (if training record[i] < reset threshold, reset mask[i] = 1, otherwise 0). ASOM visualization

[0061] SOM can be used as a tool for visualizing multidimensional data. Figure 7 shows a visualization of ASOM training of an ASOM relating five input fields (digit bitmap, even, less than 5, multiple of 3, color). The visualization shows how the ASOM can be queried to show the organization of ASOM weights during training and where the data that satisfies the query is represented for the query. The training data is provided, and each data contains a digit, followed by a (binary) flag specifying whether the digit is even, less than 5, and a multiple of 3 (in this order), and (optional) color. The ASOM is trained on the data in any preferred format. Neighbor sizes and learning rates can be gradually annealed.

[0062] Figure 7 shows the input pattern (the digits themselves are represented as a 20x20 bitmap), the reconstructed output pattern, and a flag indicating whether the network is plastic / trained with respect to the weights and has a static view. Because ASOM associates five input fields (digit bitmap, even, less than 5, multiple of 3, and color), the weight matrix is ​​decomposed into an input field weight matrix. When binary information is represented, white represents zero / false, and black represents 1 / true. When bitmaps and colormaps are represented, the colors represent their natural meanings.

[0063] Once ASOM is trained, it can organize queries and dynamically see which areas on the map best satisfy those queries. The query view is displayed in columns of the dynamic query viewer input fields, as shown in Figure 8. Each query is independent of the others and can be manipulated using sliders on their respective tabs, as shown in the screenshot in Figure 9. The queries are displayed side by side, allowing the user to visually compare areas corresponding to different queries.

[0064] To create a query, the user or automated system can specify one or more query patterns (for example, as shown in Figure 9). The intensity of the influence on each defined pattern can also be specified (as alpha / input field weight for each input field). The intensity may be binary (0 or 1) or support continuous / fuzzy mixed queries.

[0065] Each view's map can indicate the ASOM region that best corresponds to the query, and the output shows the reconstructed data that best approximates the query. By combining patterns, it is possible to ask questions such as "What are the even multiples of 3 less than 5?" or "Which numbers are shaded blue?"

[0066] The strictness of the match, in other words, the activation sensitivity of the ASOM, can be increased or decreased (as shown in the match strictness variable in Figure 9). In this embodiment, it may be desirable to decrease the match strictness if the map is completely white or the output bitmap is completely black, and it may be desirable to increase the strictness if the map is too dark or the bitmap is too blurry.

[0067] Each view has two copies of the master ASOM, one for visualizing its activity and the other for reconstructing the output. The ASOM, which calculates the output as a weighted combination of activity, requires normalized activity so that the sum is 1, while the activity map showing the ASOM should show raw, unnormalized activity to see the actual extent to which each neuron's weight satisfies the query.

[0068] Examples of ASOM include VAT (Visual / Audio / Touch), VM (Visual / Motor), VNC (Visual / NC), VAT Activity L (VAT / Location), and HC or Action Result (V1 / V2 / M / L1 / L2 / NC). Cross-modal object representation SOM

[0069] Cross-modal object representations can be learned in a System of Modulation (SOM) that associates different sensory modalities of an object. In one embodiment, it functions as a cross-modal object representation SOM that associates visual, audio, and touch inputs and learns modality-integrated representations of object types. It takes input from three SOMs that learn unimodal representations of object types: a visual object type SOM, an auditory object type SOM, and a haptic object type SOM. The signal detection process can be implemented by providing each input field of the CDZ SOM with an associated signal detection process that looks for the start of a signal in that field, and when a start occurs, triggers a qualifying trace for that field. In the case of a cross-modal object representation SOM, these signals can come from the attention systems of three distinct modalities. Learning in the cross-modal object representation SOM and its input SOMs is driven by lower-level events detected by the attention systems of three distinct modalities. For example, an event for an auditory stimulus might be a sound louder than a certain threshold.

[0070] Learning also requires some harmonization between these different signals, as it is carried out using eligibility traces. Eligibility traces (encoded by leaky integration and firing neurons) are initiated in each modality when an event is detected. If traces for two modalities are active simultaneously, learning is carried out in a type SOM for these modalities, as well as in a cross-modal object representation SOM.

[0071] If an event is detected in only one modality, connectivity to a different mode is triggered. The pattern activated in the modality registering the event is passed as input to a cross-modal object representation (SOM) that activates a modality-independent object representation. This representation is then used to reconstruct representations of other unimodal type SOMs, and as a result, patterns from missing modalities are inferred top-down.

[0072] In this model, the cross-modal object type SOM is activated by both stimulation in a single modality and stimulation in multiple modalities (provided they are harmonized). Emotional Object Association SOM

[0073] Emotional states can be associated with input stimuli. For example, emotional conditioning can be induced in a materialized agent by pairing the presentation of an object with a loud, sudden, and / or frightening noise. The next encounter with the object will then evoke a fear response. Such associations are learned in a convergence zone (CDZ) ASOM called an affect-object association ASOM. In one embodiment, this is achieved by an ASOM that associates a visual modality with a neurochemical modality (VNC SOM). This ASOM takes input from a visual object type SOM and a neurochemical SOM that holds the agent's emotional state.

[0074] Each input field of the affect object association ASOM has an associated signal detection process that searches for the start of a signal in that field, and when a start occurs, triggers a qualifying trace for that field. The signal that triggers qualifying for an object type SOM is the selection of a new prominent region in the splendor map, which can be triggered, for example, by significant movement within the field of view. The event that triggers qualifying for an affect state medium is calculated from a "phase" signal that signals a sudden change in the affect state vector, which is associated with this vector.

[0075] According to the principle of initiation-dependent learning, SOM can only learn associations between representations within its input fields if the eligibility traces of those fields are simultaneously active and exceed their respective thresholds. In this case, this principle ensures that an emotional association is learned when a newly prominent object is perceived to occur temporally concurrently with the sudden onset of a given emotion. After learning an emotional association with a given object of type O, the presentation of O as a newly prominent object automatically activates the associated emotion. This occurs through the principle of reconstructing missing inputs. Operant learning

[0076] Operant conditioning is the process by which a motor action produced by an agent in a given context is associated with a reward stimulus that arrives some time after the action is performed. The circuit that learns these associations is always active within the embodied agent, allowing it to suggest actions that yield a reward in a given context. The Action Result ASOM is a convergence zone that is built hierarchically on top of previous convergence zones. The Action Result ASOM learns the association between a perceptual context stimulus (a visual object type T1 appearing at position L1) that occurs at some given time, a motor action that is performed a little later, and a reward stimulus that occurs some time after that. The reward stimulus is associated with an object of a different type T2 appearing at a different position L2. The Action Result SOM needs to remember the representation of the perceptual context stimulus because it has disappeared by the time the reward stimulus appears. The inputs T1 and L1 to the Action Result SOM hold a copy of the previous object type triggered in the Object Type SOM, and the previous position selected in the spleness map. The inputs T2 and L2 are the currently selected spleness position and the currently active object type. Therefore, the action result SOM learns the association between the object that was remembered and the object that is currently perceived. The eligibility window can be adjusted to accommodate associated events that occur at different times. Behavior based on reconstructed memory

[0077] The behavior of all embodied agents may be affected by the reconstruction of memories. The use of a neurobehavioral modeling framework for generating and making embodied agents or avatars move as if they were alive is disclosed in U.S. Patent No. 10181213(B2), which is also assigned to the assignees of the present invention and incorporated herein by reference. Within a neurobehavioral model such as that described in U.S. Patent No. 10181213(B2), the reconstruction of inputs from different modalities can alter the internal state of the embodied agent and thus modify the agent's behavior.

[0078] By authoring emotional memories, autonomous triggering of emotional expression in embodied agents becomes possible. In customer service avatars, the experiential memory store can be used to efficiently program responses within the avatar. For example, a brand loyalty customer service avatar can be programmed to have a positive emotional response to all trademarks associated with the brand, including both visual and verbal trademarks. For instance, hearing the brand name "Soul Machines" can be associated with the feeling of "happiness," which changes the agent's neurochemical state accordingly. Hearing that word predicts a happy neurochemical state and drives the avatar to smile.

[0079] In one embodiment, a direct response to an experience or event can be "embedded" into the agent via authorable memory. For example, a memory that associates the brand "Soul Machines" with the state of paying attention / being alert, eyes wide open, i.e., the state in which the visual modality is continuously qualified, and other modalities indirectly associated with the visual modality via ASOM, receive input top-down to generate predictions.

[0080] In toy applications, embodied agents such as avatars or virtual characters can be programmed by the user to exhibit behavior. For example, a child playing with a virtual friend can "embed" in the virtual friend, through experience (presenting an object in front of the virtual friend and generating negative facial expressions and / or words) or through an interface, the memory that a particular object is unpleasant.

[0081] The personality or character of an embodied agent can be authored by authoring memories, including the agent's likes and dislikes. It is easier to develop agents with specific personalities by authoring emotional states associated with objects and / or events. Several objects can be presented and associated with emotional states. For example, an avatar can be programmed to have the personality of an animal lover by authoring file-based memories of several different animals, each associated with the emotion of "happiness." Similarly, objects can be associated with anger, sadness, or neutral emotions. Motorplan

[0082] A motor memory system can be provided to individually store motor actions and / or motor plans, which can be activated by an agent to perform the corresponding motor action(s). Examples of motor actions include, but are not limited to, pushing, dragging, and pulling. Each action is characterized by a specific spatiotemporal pattern, e.g., a sequence of proprioceptive joint positions when representing an action, or a sequence of visually perceived joint positions when representing an observed action. The time dimension can be implicitly represented by a recursive projection, and the ASOM associates the current input with its own activity in a previous computation time step.

[0083] The materialized agent may have a motor control system that allows the materialized agent to intentionally move body parts such as limbs or other effectors. Information regarding the angles of each joint can be transmitted from the skeletal model of the agent's body. The agent may have hand-eye coordination ability to reach a designated point in visual space. A self-organized map model (ASOM) can allow the agent to learn hand-eye coordination so that it can interact with the surrounding 3D virtual (or, in the case of VR / AR, real) space that changes through realistic eye movements and reaching movements. Once trained, the ASOM can be used for inverse kinematics and return joint angles when presented with a target position.

[0084] A motor action may be an individual motor movement (e.g., reaching a point in space to touch an object) or a continuous movement. For example, motor actions directed at an object, such as grasping, slapping, and punching, are continuous movements, as the agent's hand (or other effector) moves along different trajectories and / or velocities to the target object. For example, in the case of motor actions for slapping and punching, the trajectory is faster than it is to reach the object, and the trajectory may also involve pulling the hand back. The trajectories of the fingers of the hand can also be described. Grasping involves opening the fingers of the hand and then closing them. Punching and slapping involve structuring the hand into a specific shape before contact with the object.

[0085] Multiple motor actions can each be associated with a target and sequenced to create a motor plan. A system for creating a plan is described in Provisional Patent Application No. NZ752901, titled “SYSTEM FOR SEQUENCING AND PLANNING,” which is also owned by the present applicant and incorporated herein by reference.

[0086] A motor action and / or motor plan can be associated with an episode (as a WM action), an object (as an affordance of an object), or any other experience or modality. A motor action and / or motor plan can be associated with a label or other symbol that identifies the motor action and / or motor plan in the experience memory store and / or memory database and / or working memory. Examples of motor plans include playing a song on a keyboard, drawing an image on a touchscreen, or opening a door.

[0087] A motor plan may be associated with a user interface event and can trigger an event on an application or computer system with which the agent is interacting, as described in New Zealand Provisional Patent Application No. NZ744410, titled "Machine Interaction," which is incorporated herein by reference. For example, touching a target twice can be translated into a "double-click" on a user interface (in which case, a motor action of touching a "button" twice would trigger a double-click on the button on the user interface). Perceiving and retaining memories

[0088] In event-driven cognition, the biologically realistic reinforcement scheme governs which events are retained, as well as which events are perceived (and therefore communicated to other subsystems of the embodied agent).

[0089] Memory is constructed from events, but humans remember things more strongly that are unexpected. By applying this principle to embodied agents, it becomes possible to drive embodied agents by events rather than always being triggered by sensory input. This is a form of time compression that reduces the computation required for the agent to respond to an “event.” A snapshot of time related to an event can be retained based on importance, volume above a threshold, movement, novelty, contextual information, or any other suitable metric. The event contributes to memory storage by triggering a qualifying trace in modalities that create a qualifying window in which the modality is qualified to learn.

[0090] Eligibility-based learning can be used to determine which “events” the agent will hold. The occurrence of an event triggers a modality eligibility trace19. Each input channel has its own eligibility trace. For example, if a bottom-up event (e.g., a sufficiently loud sound) exists, the input channel for “sound” will be open during the eligibility window and will close after a while. An input type (modality) can be associated with a unique eligibility neuron. When an event occurs on its corresponding input channel, the neuron receives the input. The input channel is eligible for a duration during which the neuron’s voltage exceeds a threshold.

[0091] Leaky Integrator (LI) neurons can implement eligibility traces to facilitate eligibility-based learning. LI neuron activity is initiated at a specific level and decays over time. A "eligibility window" is defined while the activity of a given LI neuron exceeds a certain threshold. During this period, any associated circuit is eligible for learning. Figure 4 illustrates how modality eligibility traces 19 create an eligibility window 18 between an event trigger and a voltage threshold of a leaky integrator neuron.

[0092] Furthermore, in neurobehavioral models such as those described in U.S. Patent No. 10181213(B2), which have been transferred to the assignees of the present invention, eligibility tracing can operate on the entire network rather than on individual synapses. The steps involved in creating eligibility traces and eligibility windows for each lower-level input can be performed in the “connector” of the programming environment described in U.S. Patent No. 10181213(B2).

[0093] Eligibility windows can be used within the CDZ to control how and when learning occurs and how activity spreads through the convergence zone system. For example, in a simple perceptual convergence zone implemented by a SOM with two input fields, each input field has an associated signal detection process that looks for the start of a signal in that field, and when a start occurs, triggers an eligibility trace for that field. Learning and activity in the SOM are controlled here by several general principles that operate across all convergence zone SOMs.

[0094] The eligibility window for each input field in the convergence zone (SOM) can be adjusted based on contextual parameters. For example, certain parameters of an agent's emotional state can stretch or narrow a particular window. Thus, frustration can make a particular window shorter, while relaxation can make it longer.

[0095] In the case of a CDZ SOM that takes input directly from a perceptual or motor signal, the signal detection process associated with its input field captures the onset of the sensory or motor stimulus. If a CDZ SOM takes its input from another CDZ SOM, the signal detection process can identify the onset of a clear signal in the lower CDZ SOM. This can be read on a measure of change in the lower SOM's activity pattern, signaling that it represents something new. This measure of change can be combined with a measure of the lower SOM's entropy. If the SOM pattern can be interpreted as a probability distribution, its entropy can be measured. The change must result in a low-entropy state, which communicates that the SOM is confidently representing its input pattern. If the lower SOM is configured to learn slowly, the upper SOM will not learn until the encoding of the lower SOM's own input is sufficiently clear.

[0096] Activity can flow top-down from upper CDZ SOMs to lower CDZ SOMs via the "Top-Down Activation" field. If a lower-level SOM is highly qualified, its activity activates the upper-connected SOM, which then provides a top-down signal in real time to other connected lower-level SOMs. These can act as real-time top-down guides for the bottom-up process of calculating these inputs.

[0097] In timing-mediated eligibility-based learning, for associations to be learned within these networks, two inputs in the associated network must occur within a specific time interval from each other. Some ASOMs may only be able to learn associations between representations in those input fields if the eligibility traces for those fields are simultaneously active (exceeding their respective thresholds). This ensures that learning occurs only when new signals arrive simultaneously or with some temporal harmony, preventing the learning of associations between random or noisy signals. Thus, learning requires a window of concurrent eligibility occurrences. For example, to learn an association between a visual stimulus and a tactile stimulus, there must be a window of concurrent eligibility occurrences for the visual and tactile representations. These concurrent eligibility windows model the simultaneous occurrence of different lower-level multimodal "events." Figure 3 shows eligibility signals for different modalities. Figure 5 shows the flags and phases of a learning event. A learning event is triggered by two concurrent events (0 and 1). While the event is occurring, the agent can acquire more information, for example, by saccading or waiting for an audio sequence to finish. This delay can be implemented using leaky integrator neurons. The length of the delay can be changed by modifying the input frequency constant and / or membrane frequency constant of the leaky neuron. All indicated periods are controlled by separate leaky neurons. At the end of events, plasticity exists in 2 and 3 for "primary" SOMs, and in 4 for secondary SOMs. If there are multiple layers of CDZ in the hierarchy, this is a common learning event sequence, and the learning phase can be extended accordingly to accommodate the hierarchy.

[0098] If the eligibility trace is active for only one input field, the SOM is placed in a mode where its activity is driven only by this field. (i.e., the ASOM alpha weights for the other input fields are set to zero.) The activity in the other input fields is then reconstructed from the active SOM pattern, as described in "Inspecting Content by Top-Down Reconstruction of Weights." The reconstructed values ​​provide a useful top-down bias if the missing input field is delivered by a lower-level classification process. The reconstruction also provides a simple model of perceptual "fill-in," by which the missing associated information is inferred. It is possible for reconstructed (or predicted) inputs to arrive bottom-up while the eligibility trace for the first field is still active. In this case, the SOM learns somewhat more and strengthens the associations used to make predictions. On the other hand, if an unpredicted signal arrives in the second field while the eligibility trace for the first field is still active, new associations are learned in the SOM.

[0099] In one embodiment, a time-dependent dopamine plasticity model is implemented to determine which (and to what extent) events are learned / retained, as observed by the embodied agent. The amount of learning that takes place, i.e., the "plasticity" of the relevant system, is influenced by several factors. The level of the relevant eligibility signal(s) is one factor. Another important factor is the intensity of the concurrently occurring "reward" signal. The reward signal can be implemented as neurotransmitter levels, specifically dopamine levels. Maps such as ASOMs, stochastic SOMs, and mixtures thereof can be associated with plasticity variables that determine when the ASOM weights are updated. To prevent overtraining, plasticity can be dynamically turned on and then off at separate moments or time intervals (e.g., when a new input arrives). To prevent neurons from overtraining, the learning frequency constant can be reduced if there is a good match between the input and the winning neuron before updating the weights. Memory Integration

[0100] The experiential memory store can include two independent and competing SOMs: short-term memory (STM) and long-term memory (LTM). The STM can be configured for rapid online / fast learning, and it can be trained in a one-shot learning style with high LFC and poor topographic arrangement of data. The STM can also function as an aa buffer system for more integrated learning. Memories in the STM can be erased after each integration. The LTM can be configured for slow offline learning with a low and time-decaying learning frequency constant, resulting in good topographic grouping. The LTM is trained during memory integration (which can be expressed as "sleep" in the avatar). Training data from the STM can be simulated or replayed in the LTM SOM. During integration, the STM can activate trained units (e.g., self-organized map neurons) in a random or systematic manner to replay object types and images, which can then be provided as training data for the LTM. LTM can be trained on reconstructed data pairs or tuples and interleaved with training on new data using its own training data. Alternatively, instead of reconstructing objects using STM SOM, LTM can be trained by providing raw data files from entries in a memory database.

[0101] For example, LTM and STM object classifiers may be in certain competition for visual recognition. Both LTM and STM object classifiers can map representations of the visual space (e.g., pixels) to a common one-hot encoding of object types. If there is no sufficiently good match in the LTM, the system assumes a match in the STM. An LTM match is satisfied if its entropy is below a threshold and the winner activity is above a threshold. Thus, the STM and STM classifiers represent object types collectively and disjunctly (the STM represents object types from the last integration, and the LTM represents object types learned before the last integration). When a new object is encountered, the LTM can be checked first to see if the object exists. If the object does not exist, the object is learned in the STM.

[0102] Figure 10 shows the LTM and STM displays. The leftmost window shows the current foveal input to the visual system. The second window shows which system (STM / LTM) has a sufficiently good or better match to the foveal (bottom-up) input, indicated by the purple rectangle. Green indicates which system is affected by the top-down. The upper and lower halves of this window correspond to STM and LTM, respectively. For the remaining display, the upper half belongs to STM and the lower half belongs to LTM. The four windows arranged in a square (highlighted with solid lines) show the SOM training record with input image, output image, SOM weights, and SOM winner overlay (clockwise). The two windows on the right show the next predicted for the visual SOM replica in the sequence image. Training from memory databases

[0103] LTM can be trained during sleep using experience from a memory database. Many iterations of the memory database can be provided (randomly or systematically). language

[0104] By connecting the memory systems described herein to language systems, and by modeling the embodied agents in an environment in which they can interact, the meaning of the embodied agents can be grounded. Instead of providing a symbolic knowledge base, the embodied agents create their own meaning from the sensory inputs they receive and the actions they generate. By connecting the memory systems described herein to language systems, relevant syntactic structures abstracted away from a specific language can capture cross-linguistic generalizations. Episode memories

[0105] Agents can experience episodes that represent events in the world that can be reported in simple sentences. An episode is an event represented as a sentence-sized semantic unit centered around an action and its participants. Different objects play different “semantic roles” / “thematic roles” in an episode. For example, a WM agent is the cause or initiator of an action, and a WM patient is the target or subject of an action. An episode may involve an agent acting, perceiving actions performed by other agents, planning or imagining events, or remembering past events. Like other experiences, episodes can be stored in the experience memory store and memory database. Representations of episodes can be stored and processed in the working memory system, which processes episodes as prepared sequences / regularities encoded as distinct actions. The WM system 40 connects lower-level object / episode perception with memory, (higher-level) behavioral control, and language.

[0106] Figure 11 shows a working memory system (WM system) 40 configured to process and store episodes. The WM system 40 includes WM episodes 42 and WM individuals 41. WM individuals 41 define the individuals who play the main role in the episode. WM episode 42 includes all elements that make up the episode, including WM individuals and actions. In a simple example of WM episode 42 including individual WM agents and WM patients, the WM agent, WM patient, and WM actions are processed sequentially to fill the WM episode.

[0107] The individual store / medium 46 can be used to store WM individuals and determine whether an individual is new or has been previously focused on. The individual store / medium can be implemented as a SOM or ASOM, where new individuals are stored in the weights of newly recruited neurons, and individuals that have been previously focused on update the neurons representing those individuals. In one embodiment, the individual store / medium is a convergence zone of a CDZ that stores unique combinations of individual attributes such as location, number, and traits as distinct individuals. The ASOM preferably has a high learning rate and near-zero neighborhood size, can learn individuals immediately (one-shot), and does not have topographic organization (so that representations of different individuals do not influence each other). Traits of different individuals are stored in the weights of different neurons, and for that purpose, if the activity of the winning neuron falls below the novelty threshold, a new unused neuron is recruited; otherwise, the winner's weight is updated.

[0108] Location, number, and characteristics arrive sequentially from the individual buffer 48, one at a time. The individual store / medium 46 system is constantly queried with a non-zero alpha for already filled components, and therefore it can predict the number and characteristics based on location, for example, if an individual has recently been seen at that location. However, the system's plasticity is only turned on when the location-number-characteristic sequence is successfully completed. The individual store / medium 46 system then goes through its learning cycle, and once it is finished, it stays there for a while, allowing the individual (along with its old / new state) to be copied to the respective agent / patient buffer in the episode buffer.

[0109] The episode store / medium 47 stores WM episodes. The episode store / medium can be implemented as a SOM or ASOM trained on combinations of individuals and actions. In one embodiment, the episode store / medium is a convergence zone of CDZ that stores unique combinations of episode elements. The episode store / medium 47 can be implemented as an ASOM having three input fields, an agent, a patient, and an action, each taking input from a WM episode slot. The mixing coefficient (alpha) of the input fields is non-zero only when the inputs to the input fields have been successfully processed. This means that as the input fields are gradually filled, the ASOM delivers predictions about the remaining input fields, for example, which episodes this agent typically participates in.

[0110] The individual buffer 48 sequentially retrieves the attributes of the individual. When the sequence is complete (all buffer hold gates are closed), the plasticity of the episode store / medium 47 is turned on, and the episode store / medium 47 can store this particular combination of position, number, and characteristics as a new individual (or update the one that has been given attention again). When the episode store / medium 47 finishes its processing, the entire cycle starts again.

[0111] The episode buffer retrieves the elements of the episode sequentially. The plasticity of the episode store / media system is turned on only when the episode sequence has completed successfully. This ensures that incorrect representations are not learned when the attention mechanism makes incorrect predictions about episode participants. Reinforcement learning components

[0112] If a perceived episode yields a specific reward to the agent, it can be associated with the episode in the episode store / medium 47 as an additional input field. During episode perception, the episode store / medium 47, which has a zero ASOM alpha weight for rewards, yields a prediction of the expected reward associated with the currently perceived episode. During action execution, the reward input can be used to prime the medium to preferentially activate episodes associated with a specific value of reward. emotion

[0113] If a perceived episode is associated with a specific felt emotion or affect value, it can be associated with the episode in the episode store / medium 47 as an additional input field. During episode perception, the episode store / medium 47 with zero ASOM alpha weight for emotion will bring forth the emotion associated with the predicted episode. During action execution, the affect value can be used to prime the medium to preferentially activate episodes associated with similar emotions. Details and transformation forms of the materialized agent Top-down and bottom-up neurobehavioral modeling

[0114] Embodiments of the present invention improve artificial intelligence by combining lower-level modeling capable of emerging behavior with higher-level abstract models that, while lacking a strong biological foundation, are faster and more effective for a given task. An example of an embodied agent having an architecture capable of emerging behavior is disclosed in U.S. Patent No. 10181213(B2), which is also assigned to assignees of the present invention and incorporated herein by reference. An advanced modular programming environment enables a top-down cognitive architecture having interconnected higher-level “black boxes” (modules 10). Each “black box” ideally contains a set of interconnected biologically valid lower-level models, but can readily include abstract rules or logical statements, access to a knowledge base / database, a conversational engine, input processing using conventional machine learning neural networks, or any other computational techniques. The inputs and outputs of each module 12 are exposed as module “variables” (and thus animation parameters) that can be used to drive behavior. Connectors communicate variables between modules 12. In its simplest form, the connector copies the value of one variable to another module 12 at each time step. These higher-level symbolic processes are integrated with behaviors emerging from the model of lower-level neural circuits. The emerging behaviors interact with the higher-level processes in a natural way. The circuits performing the computations operate in parallel and continuously, without any central point of control. The programming environment can hardcode this principle by not allowing any single control script to execute a series of instructions on module 12. The programming environment supports the control of cognition and behavior through a set of more neurally plausible distributed mechanisms.

[0115] A leakage integrator has three main parameters that control the timing: IFC, MFC, and voltage threshold. When modifying parameters to control the timing, the simplest is to adjust the MFC, increasing it for faster decay, and vice versa. A typical voltage threshold for a CDZ may be 0.1. emotion

[0116] Emotions are modeled as coordinated brain-body states, comprising experienced or felt components and behavioral responses. A neuroscience-based approach models behavioral circuits modulated by physiological parameters. Physiological regulation alters the interaction of sensory, cognitive, and motor states. Virtual neurotransmitters are produced in response to stimuli, which can map to emotions and guide behavioral responses. For example, a "threatening stimulus" triggers the release of virtual norepinephrine and cortisol, which release energy for a fight-or-fight response and evoke a sense of fear. A smiling human face or gentle voice (when evaluated by some function) can trigger virtual oxytocin and dopamine, which map to a positive valence state, reducing emotions such as happiness, producing a smiling facial expression, and reducing agitated behavior. advantage

[0117] Therefore, a real-time learning network architecture is provided, where the most advanced machine learning algorithms learn offline and require large amounts of data. Agents can learn by themselves (by interacting with their environment), be taught (by the user presenting them with specific stimuli), or be completely user-controlled (by embedding memories). The architecture is a general architecture for learning, capable of learning different types of things.

[0118] Furthermore, real-time learning network architectures are not black boxes because the causes of the behaviors that emerge can be understood. It is possible to trace back through the paths that trigger the behavior. SOM can handle any form of input, e.g., a single hot vector, an RGB image, a feature vector from a deep neural network, or anything else. Moreover, the architecture can be stacked hierarchically, with lower-level inputs being integrated and further integrated with other association domains. This makes it possible to indirectly relate disparate modalities, thereby triggering complex behaviors.

[0119] Maps such as those disclosed herein allow agents to flexibly encode events and retrieve memorized information in the course of live operation. As the agent experiences the world, a map representing memorized events is presented along with new events to encode. However, while the embodied agent is experiencing this event, this same map is presented along with previously experienced portions of the event and used in a "query mode" where the agent is asked to predict the remaining portions. These predictions can therefore serve as a top-down guide to the sensorimotor process.

[0120] SOMs offer an alternative way to build HTM-type systems, but have the advantage of being topographically self-organizing and therefore having better cluster information. Unlike traditional deep networks that need to be trained slowly and offline, SOMs support rapid one-shot learning. SOMs readily support learning generalizations across the input patterns they receive. SOMs can store their memories within the weight vectors of each neuron in the map. This enables dual representation. SOM activity represents a probability distribution for multiple choices, but the content of each choice is stored in the weights of each neuron and can be reconstructed top-down.

[0121] ASOM can flexibly associate inputs from different sources / modalities and assign them dynamically changeable attention / importance. The activation flow can be reversed. ASOM supports both bottom-up (input to activation) and top-down (activation to reconstructed input) processing, as well as combinations thereof. ASOM can denoise noisy inputs, reconstruct missing parts, or return prototypes and highlighted parts where the input and prototype differ. All of this works across multiple levels of the ASOM hierarchy.

[0122] overview In one embodiment, a method for making an embodied agent move as if it were alive, comprising the steps of: receiving sensory input corresponding to a first representation of an experience in a first modality; querying an experience memory store to retrieve a second representation of an experience in a second modality; and using the second representation in the second modality to make the embodied agent move as if it were alive.

[0123] In another embodiment, a system for storing the memories of an embodied agent, comprising: an experience memory store input from experiences experienced in the course of the embodied agent's operation, wherein each experience is associated with multiple representations of the experience in different modalities, and the experience memory store stores representations of the experience in the weights of a neural network; and a memory database that stores copies of the experiences stored in the experience memory store, and stores raw data corresponding to representations of the experience in different modalities.

[0124] In another embodiment, a method for selectively storing experiences experienced by an embodied agent in the course of the agent's live operation, comprising: receiving representations of inputs from multiple input streams for receiving inputs in multiple modalities, wherein each input stream is associated with at least one condition that creates a qualification trace for the input stream; detecting simultaneous qualification traces of two or more input streams ("qualified" input streams); and storing and associating representations of inputs from qualified input streams.

[0125] In another embodiment, a method for training a SOM comprising a plurality of neurons, each neuron associated with a weight vector and a training record, the method comprising: receiving an input vector; determining whether the input vector is "new"; if the input vector is not new, selecting a first winning neuron, supporting the higher similarity between the input vector and the winning neuron, and modifying the weight vector of the first winning neuron toward the input vector; if the input vector is new, selecting a second winning neuron, supporting a neuron with a lower training record, and modifying the weight vector of the second winning neuron toward the input vector.

[0126] In another embodiment, a system implementing a method for selectively storing experiences experienced by an embodied agent in the course of the agent's operation, the system comprising: receiving representations of input from multiple input streams for receiving input in multiple modalities, wherein each input stream is associated with at least one condition that creates a qualification trace for the input stream; detecting simultaneous qualification traces of two or more input streams ("qualified" input streams); and storing and associating representations of input from qualified input streams.

Claims

1. A system for storing the memories of materialized agents, An experience memory store input from experiences previously experienced by the embodied agent during the live operation of the embodied agent, wherein each experience is associated with multiple representations of the experience in different modalities, and the experience memory store stores the representations of the experience in the weights of a neural network. A memory database that stores copies of the experience stored in the experience memory store, and which stores raw data corresponding to the representation of the experience in different modalities, A system that includes these features.

2. The system according to claim 1, wherein the embodied agent is moved to appear alive using a neurobehavioral model.

3. The system according to claim 1, wherein the plurality of modalities include one or more from the group including visual, audio, touch, motor, neurochemical, or positional modalities.

4. The system according to claim 1, wherein the experiences in the experience memory store include authored experiences.

5. The second representation in the second modality corresponds to the internal state of the embodied agent, and the embodied agent is made to move as if alive by modifying the current internal state of the embodied agent according to the extracted internal state. The system according to claim 1, optionally wherein the second representation in the second modality corresponds to the emotional state of the embodied agent.

6. A method for training a system of multiple neurons, each neuron associated with a weight vector and a training record, by a system, Steps include receiving an input vector, The steps include determining whether the input vector is "new" and If the aforementioned input vector is not new, The steps include selecting a first winning neuron, supporting a higher similarity between the input vector and the winning neuron, and modifying the weight vector of the first winning neuron toward the input vector, If the input vector is new, The steps include selecting a second winning neuron, supporting a neuron with a lower training record, and modifying the weight vector of the second winning neuron toward the input vector, Includes, The step of supporting neurons with lower training records is achieved by applying an activation mask having a mask value inversely proportional to the amount of training each neuron in the SOM has received. The training record is a weight, and the value of the training record is proportional to the amount of training received by the neuron associated with the training record. method.

7. The step of determining whether the input vector is new is, A step of determining the first winning neuron and supporting a higher similarity between the input vector and the winning neuron, A step of determining the quality of agreement between the input vector and the first winning neuron, If the matching quality falls below the matching quality threshold, the input vector is determined to be new. The method according to claim 6, including the method described in claim 6.

8. The method according to claim 7, wherein the match quality is determined as the activation of the first winning neuron according to the input vector.

9. The method according to claim 7, wherein if the input vector is not new, the learning frequency of the SOM is adjusted according to the activity of the first winning neuron.

10. The method according to claim 7, wherein the step of selecting a second winning neuron includes a step of favoring a neuron having a lower training record and a higher similarity between the input vector and the winning neuron.

11. The method according to claim 7, further comprising the step of adding random noise to the activation map of the SOM for the input vector before selecting a second winning neuron.

12. The method according to claim 7, wherein the training record of the neuron is configured to decay with SOM training and / or time.

Images