A system for neurobehavioral animation

Abstract

A computer-implemented system for animating virtual objects or digital entities. The present invention is particularly relevant to animation using biologically based models or behavioral models, particularly neurobehavioral models. A plurality of modules having computational and graphical elements are provided. The modules are arranged in a required structure, have at least one variable, and are associated with at least one connector. The connectors link the variables between the modules throughout the structure, and the modules collectively provide the neurobehavioral model. A method for controlling a digital entity in response to external stimuli is also provided.

Description

[Technical field]

[0001] The present invention relates to a system and method for simulating animatable virtual objects or digital entities, and has particular application to methods and systems for animation using biologically based or behavioral models, in particular neurobehavioural models. [Background technology]

[0002] As animation and digital technologies advance, the interface or interaction between human users and computers or digital entities has evolved significantly. Human-like machines or computer systems that can intelligently process information, interact, and present themselves in a human-like manner are desirable. This is in part because human users interact better with human-like systems and / or robots. In turn, more human-like systems may have more realistic behaviors, reactions, and animations, thus reducing perceived technology barriers, including the uncanny valley effect.

[0003] This type of animation poses several significant technical challenges. First, human-like or animal-like features need to be modeled, which is quite a challenge in itself. Then there is the challenge of taking the human-like features and using them to produce visual or graphical responses that are believable to the user or viewer. One example of a difficult response is facial expressions. If the system is one that interacts with the user (i.e., is interactive), then there is the additional challenge of processing visual and / or audio input data.

[0004] These challenges pose a technical problem: the human-like model needs to be integrated with graphics, animations and sensors in such a way that the system is flexible (it may need to be modified depending on the required use) and usable by the programmer / developer (it should be relatively intuitive or at least generally understandable to programmers), yet at the same time allows efficient compilation and execution of the system.

[0005] Existing systems do not adequately address these issues, and some known systems are described below.

[0006] Animation Program The fields of control systems and signal processing have given rise to visual programming languages ​​such as Simulink™ and VisSim™. The use of these visual systems has spread to other fields because they provide an effective way to create systems and automatically generate programming code. In a typical example, a Simulink system can be constructed by combining a series of block units (e.g., block units representing electrical components or groups of electrical components) to link inputs and outputs as desired. The system is then compiled by evaluating the block structure and system attributes, restructuring the model into a flattened structure, and sequencing the operation of the blocks. In this sense, visual design is used to create an understandable representation of the model. However, the model is run in an ordered, unified fashion. It is also known that similar visual programs make coding or circuit layout simpler.

[0007] Animation and 3D drawing programs are also known, for example Autodesk Maya™ uses a node graph architecture to represent complex 3D graphics. Autodesk Maya allows you to create and build animations across multiple different levels. You can then instruct the animations to interact with the environment. Some programs use jitter to interface between animation and functional aspects, including Max™ visual programming. In these cases, the graphics engine is substantially separate from some other program or means (such as sound for jitter), but is controlled by them. In other cases, the complexity of animation simulation is overcome by using a limited set of possible actions. For example, Havok Animation Studio™ (HAS) provides efficient character animation by using finite state machines (FSMs). Cerebella, a Virtual Human toolkit from the university of Southern California's (USCs) institute for creative technologies (ICTs), allows automatic generation of animated physical behavior based on accompanying dialogue. However, Cerebella requires detailed input about the character's mental state in order to create the appropriate animations.

[0008] Neural Model Systems Models based on neural networks, including programs such as SNNS and Emergent, provide a variety of neural network environments. In various programs, the models may provide biological-type neurons or build artificial neural networks. Effective neural networks may contain hundreds or thousands of neurons even to simulate simple models. The complexity in using large neural networks has led to attempts to build devices based on artificial intelligence (AI). Social or personal robots, such as those developed by MIT Leonardo, appear to have human-like characteristics. However, they must be programmed in a rigid and inflexible manner, usually requiring specific implementations of possible behaviors, depending on specific hardware, or being inflexible.

[0009] Artificial Intelligence Robot Neurorobotic and / or brain-based devices attempt to create human-like systems by copying brain-based functions to produce desired interactions. These models are usually very large-scale, recreating complete brain systems from low-level neurons and linking the system with a biological-like interface system. Brain-based devices are robots built to mimic the behaviors produced by the nervous system. They are usually adapted to have human-like movements and sensor arrays, but do not provide an interactive experience through human interaction. Brain-based devices are designed for a specific robot or application and usually lack broad support for a range of different operations.

[0010] In summary, known systems do not have the ability to adequately perform one or more of the following: Support for multiple models with different levels of simulation detail. · Conduct high-level and low-level simulations. · Integrating and sequencing animations and graphics as part of a simulation. Providing visual or animated output of multiple models that may jointly involve the simulated system. · Providing an environment with the flexibility required to adjust, eliminate or duplicate components of the model. · To provide an environment that is easily understandable for the modeler or developer. -Provide an animation system based on the biological nervous system. · To provide learning capabilities. Summary of the Invention [Problem to be solved by the invention]

[0011] Object of the invention It is an object of the present invention to provide a computer-implemented system or method for simulating virtual objects that may overcome or at least ameliorate one or more of the above problems, or that at least provide a useful alternative.

[0012] It is a further object of the present invention to provide a computer-implemented system or method for providing animated real or virtual objects or digital entities based on a neurobehavioral model.

[0013] It is a further object of the present invention to provide a computer-implemented system or method for describing digital entities based on a neurobehavioral model.

[0014] It is a further object of the present invention to provide avatars with increased complexity, detail, vividness, or responsiveness to stimuli, as well as to provide a system for controlling the avatars that is interactive and allows for adjustment of interaction characteristics.

[0015] Further objects of the present invention will become apparent from the following description. [Means for solving the problem]

[0016] In one aspect, the present invention addresses the technical problem of flexibly integrating real-world input stimuli into a virtual neurobehavioral model or components of a model within a machine such that the machine provides interactive animation.

[0017] In another aspect, the present invention addresses the technical problem of using a machine to integrate separate neural or neurobehavioral models of different scales.

[0018] In another aspect, the present invention addresses the technical problem of allowing a programmer to vary or change relationships between components of a model in a manner that allows a machine to be used to easily identify and implement changes in an entire neurobehavioral model, or changes in animations or interactive aspects of an animation.

[0019] In a first aspect, the invention generally provides a computer-implemented system for animating a virtual object or digital entity, the system including a plurality of modules having computational and graphical elements: The modules are arranged in the required structure, Each module has at least one variable and is associated with at least one connector; Connectors link variables between modules throughout the structure, and the modules collectively provide the neurobehavioral model.

[0020] In one embodiment, the modules are arranged in a structure, such as a hierarchical structure. In one embodiment, the hierarchy comprises a tree structure.

[0021] In one embodiment, the structure is derived from biological properties of the biological structure of the object being animated.

[0022] In one embodiment, the structure is derived from an evolutionary neural structure.

[0023] The hierarchy may be a tree structure and may depend on the properties of the object being animated. For example, the hierarchical structure may be derived from biological properties or structures present in or required by the object being animated. Thus, if the object is a human face, the structure may include a hierarchy in which a module containing computational and graphic functions related to the cornea is subordinate to (hierarchically lower than) a module related to the eye.

[0024] The hierarchical structure may additionally or alternatively be related to evolvable properties or structures of the simulated object, for example evolvable brain or neural structures.

[0025] The use of a tree structure facilitates identification of the functionality of modules within the configuration of simulated objects.

[0026] The use of connectors provides significant flexibility, allowing variables to be linked across multiple modules, creating links between modules to form complex neurobehavioral models. Connectors also aid in reducing repetition of model functionality by explicitly showing how systems are linked, providing greater efficiency in modeling and in the operation of the system.

[0027] In one embodiment, the system includes at least one module that includes an audio or graphical or visual input or stimulus and at least one module that has an audio or graphical or visual output.

[0028] In one embodiment, a portion of the system represents a brain.

[0029] In one embodiment, the graphical elements of each module may be toggled between a visible and hidden state.

[0030] In one embodiment, a module may have multiple possible graphic elements.

[0031] In one embodiment, the graphical elements of one or more modules include representations of computational elements.

[0032] Graphical elements may provide in-module support for GPUs, shaders, and other graphic tools, so that there is a simple means to generate a graphic output for each or any of the modules. For example, neuronal activity of a module can be combined into a color output, an auditory output, or a visual output, without the need to create a new module.

[0033] In one embodiment, a module represents one or more neurons.

[0034] In one embodiment, a module may represent a biological model.

[0035] In one embodiment, at least one of the modules may represent a high-level system and at least one of the modules may represent a low-level system.

[0036] In one embodiment, a variable from a module may be linked to any of multiple modules by a connector.

[0037] In one embodiment, a module may have additional modules associated with it through necessary structures that perform part of the module's operations.

[0038] In one embodiment, at least one of the modules is an association module that links the inputs and outputs of the modules through variable weights.

[0039] In one embodiment, the association module has fixed weights.

[0040] In one embodiment, the graphical elements of the module may be toggled on and off.

[0041] In one embodiment, a module may have multiple graphical elements, each with a separate graphical output.

[0042] In one embodiment, the necessary structures establish relationships between modules.

[0043] In one embodiment, multiple modules may include conversion elements.

[0044] In one embodiment, the transformation component adapts the graphical output of the modules based on the modules being linked by the required structure.

[0045] In one embodiment, at least one of the modules has a graphical input.

[0046] In one embodiment, the system has at least one graphical output.

[0047] In one embodiment, one of the modules generates a graphical output of the linked variables.

[0048] In one embodiment, one of the modules has an input from an external stimulus / stimuli.

[0049] In one embodiment, the system is capable of learning from the external stimulus / stimuli.

[0050] In one embodiment, the system provides the stimulus externally.

[0051] In one embodiment, the system is interactive with the user or the environment.

[0052] In one embodiment, one or more of the modules includes a learning or memory element.

[0053] In one embodiment, the learning element is implemented by an association element.

[0054] In one embodiment, the association element is a synaptic weight module.

[0055] In one embodiment, operation of a module is regulated by an adjustment value coupled to the module.

[0056] In one embodiment, the modulating value is neurotransmitter / neuromodulator related.

[0057] In one embodiment, if the model is time-stepped, then each module performs an action.

[0058] In one embodiment, the object may be a virtual object.

[0059] In one embodiment, the connectors may communicate using a standardized network format.

[0060] In one embodiment, the connector may communicate time-varying data.

[0061] In one embodiment, a connector may introduce timing and / or delay attributes.

[0062] In one embodiment, the timing and / or delay elements may depend on the properties of the bond or structure.

[0063] In another aspect, the invention generally provides a computer-implemented system for animating an object or digital entity, the system including a plurality of modules having computational and graphical elements: Each computational element has a module type and at least one variable and is associated with at least one connector; Connectors link variables between modules, and the linked modules together represent a graphical and computational model of the virtual object being animated.

[0064] In one embodiment, the system includes an input for receiving an auditory or visual input stimulus.

[0065] In one embodiment, the present invention may include a sensing element.

[0066] In another aspect, the invention generally provides a computer-implemented system for animating an object, the system including a number of modules: Each module has a type selected from an interface type, an animation type, and a neuron type; Each module has variables and is associated with a connector. Connectors link variables between modules, and the linked modules together represent a graphical and computational model of the object being animated.

[0067] In one embodiment, each module may be selected from a number of predefined modules.

[0068] In one embodiment, the system may include an input module of an interface type and an output module that is an animation type module.

[0069] In one embodiment, the system may include one or more learning modules.

[0070] In one embodiment, the input and / or output may include graphical or computational information.

[0071] In one embodiment, the modules are arranged to mimic biological structures.

[0072] In one embodiment, the model is a neurobehavioral model.

[0073] In yet another aspect, there is provided a method of programming an animation, the method comprising: - generating the required module structure, where each module is associated with a part of the animation, where each module may include a computational element, a graphic element, a transformation element, and a set of inputs and / or outputs, where the computational and graphic elements are associated with the part of the animation; generating a plurality of connections between a plurality of modules, the connections being between the inputs and outputs of each module; where a hierarchy of modules and multiple connections defines the system to be animated, and a model controls the system to be animated.

[0074] In one embodiment, the required structure is hierarchical.

[0075] In one embodiment, the inputs and / or outputs are variables of the module.

[0076] In one embodiment, the hierarchy and / or connections may replicate a neurobehavioral system.

[0077] In one embodiment, the hierarchy and / or connections may replicate neural circuits.

[0078] In one embodiment, the method may further include varying the coupling between the elements to vary the animation.

[0079] In one embodiment, one or more of the modules in the method may be a learning module.

[0080] In one embodiment, the method may further include adapting the learning module based on the set of inputs and / or outputs.

[0081] In one embodiment, the flexibility of the learning module may be altered.

[0082] In one embodiment, the method includes selecting each of the modules from a plurality of predefined modules or module types.

[0083] In one embodiment, the method includes adjusting the predefined modules to produce a desired operation.

[0084] In one embodiment, one or more of the required modular structures is a learning module.

[0085] In one embodiment, the method includes adapting the learning module based on the input data and then freezing the operation of the learning module.

[0086] In another aspect, the invention generally provides a computer-implemented method for animating an object or digital entity, the method comprising: providing a plurality of modules that cooperate to simulate a neurobehavioral model, each of the plurality of modules having a graphical element; processing the module such that a transformation of an anatomical feature of an object or entity results in a corresponding transformation of one or more sub-portions of the anatomical feature; Includes.

[0087] In another aspect, the invention generally provides a computer-implemented method for animating an object or digital entity, the method comprising: providing a plurality of modules that collectively provide a neurobehavioral model, each of the plurality of modules having a graphical element; processing the modules in a time-step manner to provide per-module graphical information for each time step; evaluating real-time constraints; rendering the graphical information when real-time constraints are resolved; Includes.

[0088] In one embodiment, rendering of the graphical information may occur after multiple time steps have been processed.

[0089] In another aspect, the invention generally provides a computer-implemented system for animating an object or digital entity, the system including a plurality of modules, each of which may have a computational element, a graphical element, and one or more variables; At least one of the plurality of modules generates a graphic output function; At least one of the plurality of modules is adapted to vary an appearance of the graphic output function; At least one of the multiple modules is an association module that includes weights to link input and output variables.

[0090] In one embodiment, at least one of the plurality of modules is a learning module adapted to modify future behavior of the animated virtual object or digital entity.

[0091] In one embodiment, the association module is a learning module.

[0092] In one embodiment, the plasticity of the learning module is adjustable to control the rate of learning.

[0093] In one embodiment, at least one of the plurality of modules is a learning module that has stopped learning.

[0094] In one embodiment, the association module has inputs from one or more modules that form the neurobehavioral model and outputs to one or more modules that form the graphical output.

[0095] In one embodiment, the weights of the association module are fixed.

[0096] In one embodiment, the association weights are fixed based on external data.

[0097] In one embodiment, the weights of the association module represent a graphical output.

[0098] In one embodiment, each of a plurality of the modules is an association module that renders alternative graphical outputs.

[0099] In one embodiment, each of the alternative graphical outputs may be displayed separately or may be displayed in a blended combination.

[0100] In one embodiment, the graphical output may represent a face.

[0101] In one embodiment, the alternative graphical output may represent a range of facial expressions. In one embodiment, the graphical output is a position signal for one or more of a plurality of graphical output components.

[0102] In one embodiment, the graphical output component represents muscles.

[0103] In a further aspect, the present invention may generally be described as a computer game having one or more characters as described in other aspects.

[0104] In a further aspect, the invention may be generally described as an interactive display for displaying virtual objects or digital entities as described in other aspects.

[0105] In one embodiment, the interactive display may be an advertising display.

[0106] In another aspect, the invention may generally provide a computer-implemented system for generating interactive behavior, the system including a plurality of modules having computational and graphical elements: The modules are arranged in the required structure, Each module has at least one variable and is associated with at least one connector; Connectors link variables between modules throughout the structure, and the modules collectively provide a behavioral or neurobehavioral model.

[0107] In another aspect, the invention may generally provide a computer-implemented system for generating interactive behavior, the system including a plurality of modules having computational and graphical elements: At least one of the plurality of modules receives an external stimulus; At least one of the plurality of modules provides an external output; At least one of the plurality of modules generates an association between an external stimulus and an external output; The associations influence future system behavior such that external outputs respond to changes in external stimuli.

[0108] In one embodiment, the associations provide a learning behavior to the system.

[0109] In one embodiment, at least one of the modules generates an association between a first internal stimulus and a second internal stimulus or an external output. In one embodiment, at least one of the plurality of modules has an adjustment means for adjusting the function of one of the plurality of modules.

[0110] In another aspect, the invention can generally provide a computer-implemented method for animating a virtual object or digital entity, the method comprising: instantiating a plurality of modules from a plurality of module templates; defining functions, inputs and outputs for a number of modules; defining couplings between inputs and outputs of a plurality of modules; The plurality of modules and connections form a behavioral or neurobehavioral model.

[0111] In one embodiment, at least one of the inputs and / or outputs to at least one of the plurality of modules is an external stimulus or an external output.

[0112] In one embodiment, any one or more of the multiple modules or combinations may have a visualization output.

[0113] In another aspect, the invention generally provides a computer-implemented system for generating animated virtual objects or digital entities, the system comprising: A plurality of module templates, which may have computational and graphical elements; a first description means for specifying functions and variables of one or more selected modules, each selected module being based on one of a plurality of module templates; a second description means for specifying a plurality of connections between variables of one or more selected modules; wherein one or more selected modules are combined to generate a behavioral or neurobehavioral model.

[0114] In one embodiment, at least one of the module templates is a neuron model.

[0115] In one embodiment, at least one of the module templates is a delay model.

[0116] In one embodiment, at least one of the module templates is an association model.

[0117] In one embodiment, the system further comprises a third description means for specifying relationships between the modules.

[0118] In one embodiment, the relationships are hierarchical.

[0119] In one embodiment, the structure or hierarchy may or may not represent a biological system or structure.

[0120] In one embodiment, each module is time step capable.

[0121] In a further aspect, the invention may generally provide for control of a computer-generated display, effect or avatar using a network of modules of defined functionality coupled to communicate using a format for time-varying data, the coupling introducing timing and / or delay attributes into the time-varying data depending on the placement of the modules within the network such that the response evoked by the network can be adjusted by rearranging the modules or couplings.

[0122] In a further aspect, the invention may be generally described as a computer system operable to control a digital entity in response to data defining a stimulus for the digital entity, the system including a network of functional modules of code, the network operable to receive data characterizing the stimulus and to generate data defining a response for the digital entity, the network comprising: one or more variables per functional module, the variables configured for a standardized time-based data format for the network and associated with at least one connector that transmits time-varying data between the sending and receiving variables of the module; - location reference data defined for each module to enable the location of the module to be defined relative to one or more other modules; a time adjuster operable to adjust the timing of time varying data transmitted between a transmitting variable and a receiving variable, the time varying data being dependent on a location of the module of the transmitting variable relative to a module receiving the receiving variable; one or more functional operations defined for each of a plurality of functional modules, the one or more functional operations being operable on time-varying data carried in the time-varying signals received in variables defined for the functional modules; Contains code that defines Thereby, the operation on the time-varying data received by two receiving modules receiving data transmitted from two different functional modules is adjustable according to the adjusted relative positions of the functional modules; This allows the avatar's response to be adjusted.

[0123] The time adjuster may include a set of transmission paths operable to couple the transmitting and receiving variables to each other and operable to introduce a time delay depending on the difference in module positions of the transmitting and receiving variables.

[0124] The network may include a transformer operable to allow for the combination of two or more variables, such that two transmit variables can be combined into a single receive connector.

[0125] The functional modules may include wrappers operable to parse data in a time-based format into a given format so that the code can run on data that is not in the time-based format used in the functional modules in the coupled network using a standardized time-based format.

[0126] The functional actions may have adjustable parameters to allow tailoring of the avatar's response.

[0127] The functional operations may have parameters that are adjustable depending on network parameters that propagate through the network. The propagation may start at a defined location in the network and propagate from that location, such that the parameters can adjust the functional operations depending on the location of a given module as well as the degree of propagation of the network parameters that are propagated. The network parameters may be adjustment values.

[0128] The network may be operable to receive data or input to determine the location reference data.

[0129] As used herein, data is used broadly to encompass coded information, and may encompass instances of data types and event types, and may encompass stream data.

[0130] The network may include time adjustment means independent of the relative positions of the functional modules to allow for defining a delay or time progression of time-varying data to adjust the response for the avatar to a stimulus.

[0131] The functional operations may be operable to define feature associations of time-varying data from transmitted variables of two or more functional modules. Responses for the avatar to stimuli may be adjustable by adjustments to the relative positions of the functional modules within the network and / or to adjustments to the functional operations of one or more functional modules.

[0132] The response of the avatar to stimuli as controlled by the system may be configured by the functionality of the modules, their coupling and their relative positions within the network.

[0133] The time adjustment means may include a delay introduced into the data in a time-based format.

[0134] The network may include code defining a set of connectors that couple the transformers and modules, each connector including a time adjuster operable to apply a delay to a time-varying signal.

[0135] The network may include code defining a set of transformers operable to combine time-varying data from two or more transmit variables to enable combination into a single receive variable.

[0136] The operation of a network may depend on both the functionality of the modules and the relative location of the modules within the network.

[0137] In some embodiments, the transformer does not introduce any time delay, as is found in the functional modules.

[0138] In one embodiment, the modules are selected and / or configured by an operator.

[0139] In one embodiment, the system may include a configuration interface operable to receive adjustments to the location reference data.

[0140] The configuration interface may be operable to enable a combination of selected modules to configure a network, whereby control of the system and / or avatar responses may be configured. The configuration may be operable to display a representation of the relative positions of the functional modules and the combination of modules. The configuration may be operable to display a representation of the network. The configuration may be operable to display an avatar. A configuration may be selected to allow a user to view the network as it operates and / or adjust module positions and / or module selection.

[0141] In some embodiments, the data characterizing the stimulus is received from a camera. In some embodiments, the system may be operable to control a single avatar, or multiple avatars, individually or collectively. In other embodiments, the system may be provided within the code of an application, such as a game, and the data characterizing the stimulus may be received within the game.

[0142] One or more systems may be used to generate multiple characters represented by avatars, where similar networks are configured in different ways to diversify the characteristic responses of the avatars. This may be used to provide a set of avatars or digital entities with different characteristic responses. Different configurations may be achieved by changing parameter settings. For example, different personalities may be characterized by levels of sensitivity or response to neurotransmitters, neuromodulators or other signals in the model (e.g., by changing threshold variables). Different configurations may also be achieved by adapting the topology or layout of the system to generate different types of structures in the neurobehavioral model. The topology or layout may be changed by adjusting the coupling between modules, the functions of the modules or structures, or the relationships between modules.

[0143] Embodiments of the present invention allow for a range of different types of code to be included within functional modules that are coupled together via connectors that use a standardized time-based format so that a variety of functional codes can be included within the same network.

[0144] Another aspect of the present invention is a system for rendering facial graphics, comprising: a graphics rendering layer that receives muscle actuation / position data defining degrees of actuation of a set of facial animation muscles and generates graphics image data; a muscle activation / integration layer that receives nerve activation data defining a degree of nerve activation for a given set of animation nerves, and generates muscle activation data for the set of activated muscles defined for the muscle activation layer; a neural activation layer receiving facial expression data defining facial expressions, the neural activation layer generating neural activation data defining combinations of animation nerves to be activated and defining the degree of activation for each nerve; The present invention provides a system including:

[0145] Each layer may contain data defining the properties of nerves, muscles and skin / fat etc. The muscle layer / graphics rendering layer receives stimulation data and generates feedback data.

[0146] In another aspect, the invention may generally provide a computer system operable to control a digital entity in response to an external stimulus, the system including a network of functional modules of code, the network operable to receive data characterizing the stimulus and to generate data defining a response for the digital entity, the network comprising: One or more variables for each functional module; a structure allowing the location of a module to be defined relative to one or more other modules; one or more connectors, where one or more variables are associated with at least one connector that transmits data between the variables of the module; Including, The connectors are selectively adjustable to couple various modules together, thereby altering or adjusting the behavior of the digital entity in response to external stimuli.

[0147] In another aspect, the invention may generally provide a computer programmed or operable to implement a system according to any one of the preceding embodiments.

[0148] In another aspect, the invention may generally provide one or more computer-readable media storing computer usable instructions that, when used by a computing device, cause the computing device to implement a system of any one of the aforementioned embodiments.

[0149] In another aspect, the invention may generally provide a method of controlling a digital entity in response to an external stimulus, the method comprising: Receiving data characterizing the stimulus; processing the data in a plurality of interconnected modules which collectively represent a neurobehavioral model and providing an output defining a response of the digital entity to an external stimulus; modifying couplings between one or more modules or modifying variables in one or more modules in response to the output; Includes.

[0150] In a further aspect, the invention may generally provide a computing device operable to perform a method for controlling a digital entity.

[0151] In a further aspect, the invention may generally provide one or more computer-readable media storing computer usable instructions that, when used by a computing device, cause the computing device to perform a method for controlling a digital entity.

[0152] Any of the above-mentioned embodiments may relate to any of the above-mentioned aspects.

[0153] According to a further aspect, the present invention provides a method and system substantially as herein described with reference to the accompanying drawings.

[0154] Further aspects of the invention, which is to be considered in all its novel aspects, will become apparent from the following description, given by way of example of possible embodiments thereof.

[0155] Throughout this specification, any discussion of prior art should in no way be taken as an admission that such prior art is widely known or forms part of the common general knowledge in the art. [Brief description of the drawings]

[0156] [Figure 1a] 1 illustrates an embodiment of the present invention in which a model of the brain is displayed. [Figure 1b] A schematic diagram of a computer system for implementing the system is shown. [Diagram 2] 1 illustrates an embodiment of the present invention in which a life-like visualization is displayed. [Diagram 3] 1 illustrates an embodiment of the present invention in which multiple modules are linked. [Figure 4] 1 shows an embodiment of the present invention in which a number of modules are arranged in a folder structure. [Diagram 5] 1 illustrates an embodiment of the present invention in which multiple different models are linked. [Figure 6] 1 illustrates an embodiment of an eye blink module with links to a graphics system and a computing system. [Figure 7] 1 shows a schematic diagram of an embodiment of a module having a computational part, a graphic part and a transformation part. [Figure 8] 1 shows a schematic diagram of an embodiment of a system including a complex graphical output. [Figure 9] FIG. 1 shows a schematic diagram of an embodiment of a system surrounding the cortex. [Figure 10] 1 illustrates a system for controlling an avatar in response to data defining a stimulus. [Figure 11] 10, but shows a system with an exemplary emotional response module added to the network. [Figure 12] 12 shows the system of FIG. 11 in which parameters defining aspects of the functional operation of the modules are adjusted. [Figure 13] 13 shows a system similar to that of FIG. 12, where the network has an additional voice recognition module for multimodal recognition of face and voice. [Figure 14] 13 shows a system similar to that of FIG. 12, where the network has an additional voice recognition module for multimodal recognition of face and voice. [Figure 15]We present a model of a neurobehavioral system that shows how different neural and computational elements can be combined. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0157] Description of One or More Embodiments The invention described herein is implemented using a computer. Referring to FIG. 1b, the computer has an input means 201 for inputting information, a processor means 202, and an output means 203. The processor means for processing information may communicate with a storage means 204 for storing or retrieving information. The input or stimuli may come from real-world stimuli, including, for example, input from one or more cameras, electromagnetic transducers, audio transducers, keyboards or other known systems. Other stimuli include graphical user interfaces, hardware consoles, stream data, as well as data from cloud computers, computer indexes, the World Wide Web, or various sensors. The output means transmits signals to a display unit or another machine (e.g., a robot). The storage means may be a computer readable medium suitable for storing code executable on a processor. Alternatively, the model or a part thereof may be a circuit. An embodiment of the present invention includes a model having application in any one or more of the following forms: games, consoles, vending machines and advertising, mobile devices, and cloud computing devices.

[0158] In one embodiment of the present invention, biological behavior is simulated by a biological model that provides a graphical output. Graphical output may refer to any form of visual or presented output. For example, brain processes that give rise to behavior and social learning are used to animate a lifelike facial model that can interact with the user. In another embodiment of the present invention, the model may be applied to interactive animation. The animation may incorporate a multi-scale computational model of the underlying neural systems involved in interactive behavior and learning. Each computational unit or computational module may act as a self-contained black box that can implement a range of models at any scale (e.g., from a single neuron to a network). Thus, the modules can be linked to generate networks or structures that form the models.

[0159] Neurobehavioral models use underlying neural pathways or circuits to generate behavior. The generated neural circuits can range in complexity from relatively simple feedback loops or neural nets to complex representations of biological systems. Thus, virtual objects or digital entities encompass both large-scale models of humans or animals, such as an infant's face, as well as any other models that are represented or available in a virtual or computer-generated or implemented environment. In some cases, objects or entities may not be complete, and they may be limited to a portion of the entity (e.g., a body part such as a hand or face), especially when a complete model is not required. Avatars or other representations of people or objects are encompassed in the definition of a digital entity or virtual object. In some embodiments, the characteristics or behavior of a digital entity or virtual object may be variable via the neurobehavioral model. The system animates the digital entity or virtual object to enable realistic movements or changes of the entity or object.

[0160] Animation may synthesize or reproduce behavior and present this behavior through advanced 3D computer graphics models. In a broad sense, the model may provide a behavioral system that can adapt to external stimuli, where external stimuli refer to stimuli other than the model's internal stimuli. For example, an embodiment may interact with humans through a screen interface or may be implemented as a robot. This functionality may be achieved by a neuro-type system or by a mix of a neuro-type system and a functional replacement for the nervous system. An embodiment of the system may be referred to as self-animated, since animation is performed from external stimuli using learned methods without the need for animation intervention.

[0161] In one embodiment of the present invention, the graphic / animation elements of the model with the computational elements of the model are linked in a required structure, preferably a hierarchical structure. The structure allows sections of code to be included or grouped, meaning that sections can be duplicated or moved as a group of components. The structure may include a subordinate structure including a tree-like element. In alternative arrangements, the hierarchical structure may be implemented in another form to generate the required structure. In one embodiment, multiple hierarchies may be used. An important feature of the required structure is that it provides further links between modules, said links being focused on relationships or physical or pseudo-physical placement. In this way, the required structure provides a skeletal or relational structure for each of the modules in the model. In a preferred embodiment, the required structure is arranged hierarchically to make the structure easily visible and understandable. This allows for improved depiction of the model and allows the modeler to build the model more efficiently since modules including graphic and computational elements are related in a clear and constructable manner.

[0162] An embodiment of the present invention may include a model defined by a series of modules in a hierarchical structure. This may be similar to the way a physical element can be decomposed into its composite or component parts. Each module may have zero, one or more subordinate modules. The modules may form a tree-like structure. This structure is used for or related to the graphic structure, but also includes computational elements. The computational elements may be defined in a separate but equally necessary structure / hierarchical structure. An element may refer to a section, sub-module or part of code, or a link to code to perform a function. Having separate code elements allows for separation of control of the various functionalities in each module. Preferably, a module may include both computational and graphic elements (or either one). In a preferred embodiment, each module is capable of containing each element and only requires that the element be activated. In this way, the structure of the model can be clearly observed when looking at the hierarchy as well as the relationships between the modules, and their computational and graphic elements are clear. Thus, the model may provide an improved method of generating neurobehavioral or psychobehavioural animations. In some embodiments, more elements may be present to provide additional functionality or to separate modular structures. Sensing elements may be included in one, multiple, or all modules to allow input from internal or external stimuli.

[0163] A graphic element typically contains geometry, shader and texture information or code, whose characteristics can be combined or modified by external modules. Shaders and textures may be used in a general purpose GPU (GPGPU) sense for computation. A typical implementation of a graphic element may be for a virtual face. The face geometry, textures and shaders may be kept in a directory called "face". The face directory may also contain computation elements associated with the face. In this way, the graphic element and computation elements are contained within a single module in the required structure, but are also separated to allow for management and updating or linking. In particular, various graphic elements may be run, for example to show the running of a neural net or the movement of the face. For example, a computation element may feed muscle activation variables from a facial nucleus module to a shader deformation module or an animation deformation module, which may in turn You can deform the vertices of the facial geometry, Modify the mapping of loaded texture data (e.g., to change the appearance of the skin based on blood flow-induced facial expressions), · You may modify shading calculations based on externally calculated distortion information bound to the shader.

[0164] The required structure is complemented by couplings between multiple modules. These couplings or links help to control the complex dynamics and interrelationships between the computational system and the animation requirements. Couplings may link between inputs and outputs (variables) of any module or element in the model, including both graphic and computational modules. The communication and flexibility between the graphical and computational aspects of the model allows the designer / programmer or user to efficiently generate highly complex models. There is no need to replicate features or behaviors in separate or lightly linked graphical and computational sections of the model. In one embodiment of the present invention, inputs and outputs may be preferentially coupled to or routed through higher level modules, such that the branches of the hierarchy may be substantially self-contained. Thus, most of the couplings may be made to the higher level modules to avoid over-complexity of the internal modules. Couplings and other model features provide a means to modulate signals within the system. Modulation of signals allows behavior to be trained and training is efficient, since training is independent of the details of the model. For example, a neurotransmitter may be implemented as a connection to multiple models and its value may be altered to tailor the model or the model's behavior.

[0165] Couplings may be formed between the graphical and computational elements of a module, and these couplings provide a means for generating complex human-like simulations based on complex biological models. The couplings may provide an association between a first variable and a second variable, where the variables are inputs and / or outputs of the module. This improves on prior art systems that allow the generation of neural models but have limited graphics or animations and limit the interface / interfaces between them. By combining the graphical and computational elements, feedback loops and relationships between animations and the underlying model can be controlled and / or depicted. This also allows for more efficient model updates, since the inherent relationships may be visible, including in real time, as well as during updates or optimizations.

[0166] In one embodiment of the invention, each module is coupled to other modules to form a network or required structure. In this embodiment, the variables are coupled (shown as points on modules with lines connecting them) by couplings (sometimes called transmission lines) that may introduce delays that depend on or represent distances in the network between the mutually coupled modules. In some embodiments, the couplings determine the delays using or in response to location reference data associated with the coupled modules. In other embodiments, the delays are introduced within the modules or in separate delay modules. In some embodiments, a time advance is used instead of a delay, or there may be no delay. The couplings may transmit time-based data in the form of time signals between modules. The modules operate on these signals along with other signals to generate responses that are used to control, for example, an avatar or digital entity that is displayed to the user on a screen or other digital display device. Both the relative timing and operation of the received time-based signals affect the output of the modules. Thus, the response or characteristics of said response of a digital entity, avatar or virtual object may be influenced by any one or more of the selection of modules, the selection of functional operation of modules, the placement of modules and / or their relative positions within the network, and the selection of couplings between modules.

[0167] As used herein, the term connector or transmission line may be any communication line suitable for coupling two or more variables and may include interfaces for objects. Timing and / or delay attributes may be affected by coupling between modules. For example, in an embodiment where variables transition from sending to receiving variables at each time step, the communication of data will be delayed if there is an intervening module between the sending and receiving variables. In other embodiments, the coupling itself may have a timing or delay component. Preferably, a standardized network or communication format is used so that all modules can communicate between variables. This may require wrappers or initialization of the modules that define how code inside the modules creates a standardized network format. In alternative embodiments, the location of the modules may affect the timing, as well as the visual distance or other location reference data of the modules.

[0168] An exemplary model of the brain and facial features is shown in FIG. 1. The model may include biological, computational and graphical elements. Preferably, the computational and graphical elements may be based generally or substantially on biological systems. In one embodiment, the biological modeling architecture allows a series of low-level modules to be combined or assembled into groups that are later combined to form higher-level components. This may follow or be derived from an evolutionary hierarchical structure or an evolutionary neural structure, where simple basic modules are linked and combined to give rise to complex overall features. The basic modules may provide the core functionality of the module, and the higher level modules provide additional functionality that is combined with this more basic system. Thus, the biological modeling architecture is used to build a biology-based animation system. An advantage of the above system is that complex animated systems can be built by building multiple separate low-level modules, and the connections between the low-level modules give the model human-like functions or animal-like capabilities.

[0169] FIG. 1a shows a complete view of the model showing a representation of the brain including some surrounding features. The model may include sub-models of the nervous system and neuroanatomical structures, including systems based on scientific models and neural network systems. In particular, known types of biological neural networks may be used. The structure allows visualization of the internal processes generated by the computational models that give rise to the behavior. The structure also provides a depictable or understandable form of interaction between the user and the system. The models or modules may be driven by theoretical models, empirical models driven by data, a combination of these, or simplified models. In some embodiments, the interactive nature of the model allows modifying the behavior of the avatar or animation to build desired behaviors or to test behavioral patterns or biological effects.

[0170] The generation of avatars, digital entities or animations (such as those in FIG. 1a) requires modeling methodologies for building, visualizing and animating nervous systems. A novel model environment and method for neural models is disclosed, which may be referred to as a brain language (BL). The BL allows users to generate animations and real-time visualizations from biologically based neural network models, allowing the effects of the model to be seen in an interactive context. For example, FIG. 1 shows an image of the model's brain and eyes 21, its sections 22, and the variables, inputs or outputs 23, 24, 25. Such a visual environment is not only suitable for model generation, but is also ideal for model development and model visualization. A user interface can be used for visualization to allow tuning of the network or to allow the user to receive configuration inputs. The model receives inputs and provides outputs, visually, audibly or graphically, using cameras, microphones or any other sensors as needed. Different forms of input may require different input modules with appropriate wrappers to incorporate the data into the model.

[0171] The BL modeling environment provides two-way communication between the user and the model. In an embodiment, the model may interface with the user through visual and / or auditory communication. This means that the model may generate sounds, change direction or position, and react to the user doing the same, and preferably these actions should be realistic and human-like. In one example, the model may cry if the user is not looking in the direction of the model. Alternatively, the model may monitor and react to sounds or actions in its environment. In further embodiments, sounds or actions in the environment may affect the operation of the model over time. Animation and interaction between the environment (through sensors) and the neural model are possible, as the modeling environment provides a highly interconnected structure for complex systems. This provides a means to test, improve and optimize the model.

[0172] FIG. 2 shows an animation output, which is a 3D display of an infant's face and upper body. The output display may display the results of the model as well as affect the model. For example, the system can analyze video and audio inputs in real time and react to the behavior of the caregiver or peer using behavioral models. Similarly, the model may be affected by the direction the animation is looking as well as any external audio or actions. The external face may be represented using biomechanical information or modeling. In animation, this is usually based on muscle shapes. In alternative embodiments, the output may be part of a robot, cartoon figure or other means. These may not directly resemble human or human-like features, but may share human-like actions or responses. In embodiments based on animal-like or human-like features, the biological basis of the model may allow or require realistic modeling restrictions that produce a more realistic model. Some variables, inputs or outputs 32 as well as animated outputs 31 may be shown to improve understanding of the model.

[0173] The system may be capable of both depicting and animating digital entities. Representation allows the digital entities to be displayed by the structure and arrangement of parts of the model. This allows the user to build models efficiently because the design and animation are tightly coupled, instead of requiring the creation of separate neural and animation models and then retroactively connecting them. The representation of the model may include run-time data and / or a representation of what the system is and how the parts of the system are connected. The animation of the digital entities is tightly coupled to this representation, but adds computational and graphical information related to how the system runs and how each part of the system operates at a time step. Tight coupling of models is a feature generated by modules that contain and directly link graphical, computational and / or transformation elements such that each module forms a segment of the overall model. This allows component-level modules to be built into a cohesive whole or combined to form a coupled modular structure.

[0174] In one embodiment, the module may include a muscle level component for facial animation, which may be driven by a neurobehavioral model and may have a graphic element related to the muscles and a computational element related to the muscle actuation. The muscle component may receive input from a model that presents a preferred position or action to be taken. A neural network pattern generator may receive input or expected output from a collection of similar muscle components and combine these to form a cohesive output effect for a larger muscle or muscle region. Due to the low level of control of the graphic output, very complex facial expressions can be formed. This is because the model is not simply attempting to combine a set of possible expressions or to match a mesh of data points to the entire face, but instead to construct facial expressions based on the anatomical or biological systems of animals or humans. Other embodiments described later may provide cohesive facial expressions through a combination of outputs or expressions and finite element elasticity.

[0175] Low level control over the model's graphical and computational elements also provides the ability to explore aspects of the model at a range of levels of detail. For example, if a particular muscle movement is important, the model may be limited to showing this while keeping the rest of the model computed or running. Similarly, the model may display both output graphic animations and outputs related to the model's computations, including the graphical output of this computation. For example, a human infant model may be used to explore the effect of dopamine on blink rate. The primary graphical output may be the infant's face or head and its facial movements. However, plots of dopamine levels in the infant may also be visualized to make a comparison similar to that shown in FIG. 2. In a second example, a human infant model may be used to interact with a user to model the effect of dopamine on reward learning of a particular behavior. For example, if the infant makes a particular facial expression, the user responds in a positive way, and then the learning effect of dopamine modulates the plasticity measures, making this facial expression more likely to be repeated. The user may see changes in the infant's facial behavior, and may also visualize changes in the synaptic weights of the striatal neural network.

[0176] In one embodiment, visualization of simulated neural circuits may allow a user to see the neural circuits that give rise to behavior in a behavioral context at any given time, or in a more schematic view. A feature of the model is to look graphically under the skin to examine the activity of neural circuits that contribute to facial muscle activation. The range of viewing styles available in BL allows the user to optionally view various parts of the model in their neuroanatomical context, as well as providing a more traditional "numerically focused" view that may be better suited to more abstract models and to live model parameter changes.

[0177] Visualization can be achieved by adding graphical or visual elements to the dopamine variables or connections derived from the dopamine variables of the appropriate modules in the model. The user may then wish to examine the effect of a drug on the dopamine / reward system. This may involve adding connectors derived from the drug to a module of the neurobehavioral model. The user may wish to examine how this affects the operation of a part of the nervous system. Again, this can be achieved by generating or activating graphical or visual elements associated with that part of the system, which may be activated at multiple levels, from facial components to individual neuronal modules. This is possible because the simulation is built from a combination of modules with the required structure, with computational and graphical elements such that both computational or data-based processing and graphical processing can be inspected, depicted and adapted individually or separably. The module-based approach also allows for the addition of further detail in either the computational or display elements by introducing further modules into the required structure. In this way, the state of the model generating, for example, facial behavior can be visualized by graphs and schematics, or by examining activity mapped to the underlying neuroanatomy.

[0178] Animation / Graphics FIG. 2 shows an animation with a human-like face. The generation and animation of realistic faces and facial expressions may be achieved by the use of a model. A neural control system is preferred as a generative model for facial animation because it builds facial movements from facial building blocks. This may help generate more consistent overall expressions in digital entities or virtual objects. The neural control of facial movements requires the use of multiple parallel systems, including anatomically and functionally distinct voluntary and emotional systems, all the way up to the facial nucleus. The ability to control facial animation or expressions based on connections to the nervous system provides a means to create realistic animations and configure and optimize them to make them more human-like. The facial animation of the model may use a neuroanatomical model based on the architecture of the facial movement system. It may receive inputs from other modules associated with the model (although it would be preferable to base it on known scientific models). In an embodiment of the system, the face or other graphic object may form separate parts of the structure or separate structures to focus on the graphic requirements of a realistic face. This facial structure would then be coupled or linked to a neurobehavioral model for control.

[0179] In one embodiment of the present invention, a complex graphical output such as a face may be formed by a series of modules, either containing only or mainly containing graphical data. In this way, the face may be further independent of computational aspects to allow for modification of the details of the face (or other graphical image). The face may be arranged as a set of modules, with a first module representing the face and then multiple modules in the required structure representing sub-features. In a first example, the modules may represent only the surface of the face, or in a second example, the modules may represent the face as well as the muscles or tissue elements behind the face. Facial features may be obtained as described above, where a series of facial expressions are calculated, recorded and depicted. In one embodiment, facial expressions may be fused into a model to generate a composite expression. The advantage of using fused facial expressions is that this ensures that the expression is complete and that all of the necessary muscles are used. For example, the asymmetric nature of a smile, where individual muscles are instructed rather than working together in an organized smile pattern, may distinguish a person's forced smile from a genuine smile.

[0180] Referring now to FIG. 8, an embodiment of the present invention illustrates facial expressions in a face controlled via multiple modules. The face is represented by multiple muscle shapes 81 or facial features that represent the surface of the face. The action and graphic representation of these muscle shapes are provided by a series of facial expression modules 84, 85, 86. The facial expression modules may have predefined weights for the muscle shapes 81 so that when triggered, they provide weights, intensities or inputs to the face to generate the appropriate facial expression. The predefined weights may be obtained from previous data capture. For example, the facial expression modules may relate to a frown 84, a smile 85 and an angry expression 86. The facial expressions may be controlled by one or more pattern generators that are used as facial expression generators 87, 88, 89. The facial expression generators 87-89 provide a means to blend and shape multiple facial expression modules. For example, a parent may require an angry frown which may be achieved by blending an angry expression face with a frown face. In other examples, the facial expressions are more distinct, for example combining a smile and a surprise to generate an overall happy surprised expression. The facial expression generator may include pre-trained response curves based on those input variables.

[0181] The facial expression generators may be linked to the neurobehavioral model, and in particular to modules that target emotions. For example, they may be coupled to modules for punishment 90 or reward 91. The facial expression generators may receive input from any one or more of the system's distinct emotions and compose an appropriate blend of facial expressions for output. To provide the necessary structure and support the organization of the system, the system may be included under an overall structure element 95. In a preferred embodiment, the system includes one or more learning elements associated with the facial expression system. For example, the learning elements may modify the weights of the expression generator so that a model that has been rewarded for a long time for making a particular expression has a stronger weight for the rewarded expression and therefore exhibits that expression more frequently. The model may also learn by conditioning. For example, after conditioning, an association module that associates a stimulus-induced sensory activity with an emotional trigger of a behavior will trigger the behavior even when exposed to the stimulus alone. This is similar to "classical conditioning" or "Pavlovian conditioning".

[0182] Graphical output or animation may be constructed using a brainstem or cortical pattern generator modeled using a biologically plausible recurrent neural network model formed as a module, generating activity that is resolved in the facial nucleus before being output as animation weights. The graphic output of the model as shown (as well as the specific facial model) is designed to be driven by muscle activation resulting from simulated motor neurons. The motor neurons may be included in a separate module or may form part of the graphic portion of the module. The geometric diversity of facial expressions is generated by modeling the effect of individual muscle activation as well as significantly nonlinear combined activation. Graphical animation may use procedures including scanning, geometric modeling and biomechanical simulation.

[0183] A variety of facial expressions may be generated using a comprehensive biomechanical simulation of the activation of individual muscles. This provides the advantage that the graphical output is a combination of each of an array of individual features, but forming a single cohesive and flexible whole. The system may also allow certain sections to be turned on and off to allow viewing of obscured areas or to focus on specific features. If the model included in the system simulates the human brain, or parts of it, then turning off certain sections of the computational model can make the system behave as if it had suffered brain damage. This may be referred to as an artificial lesion.

[0184] The system may also allow facial expressions to be artistically modeled to allow the construction of stylized characters or cartoons that share human-like characteristics. An advantage of using a physics-based model is compatibility for future robotic implementations of the model (e.g., in humanoid robots). Facial expressions may be based on a limited set of facial biomechanics, but in a preferred embodiment, the structure is closely matched to facial anatomy. The digital entity may include an avatar as well as a display or audiovisual output including lighting, cartoon characters, colors, or other output or display known to the viewer.

[0185] The system may include models of deep and superficial fat layers, deep and superficial muscles, fascia and / or connective tissue. Facial deformation or facial movement effects may be obtained through the use of large deformation finite element elasticity, which is used to deform the face from a resting position via activation of simulated muscles. Each muscle's action may be simulated, alone and in combination with muscles typically activated simultaneously, to generate piecewise linear facial expression variations or facial movement patterns. In one embodiment, this calculated facial geometry is combined with the model to generate real-time facial animation with realistic deformations and skin movements. However, other methods may be developed to generate calculated or process geometry on the fly. In one embodiment of the present invention, the graphical output may be used to drive a robotic face or similar physical representation. Along with the physical representation of the face, the graphical output may be able to display further details about the graphical output, such as tension or stretch or pressure.

[0186] Learning elements The model also allows for learning and / or episodic memory elements. These elements may be implemented in the form of modules that store a set of data values ​​or weights. The weights may be associated with a neural network. In one embodiment, at least some of the weights are synapse-like weights. The weights may change or adapt to changes in the model's environment or inputs. For example, they may respond to dopamine levels, where higher levels may indicate a more plastic state. In this way, the model's neural network automatically adapts and the results may change over time. The dynamics of social learning are key in developing realistic interactions, since the party or user interacting with the infant expects the infant to change its response when provided with positive or negative reinforcement feedback. The learning module may form part of the computational portion of the module and may, if desired, be displayed in graphical form as described above.

[0187] The learning module may be a federated learning element or an association engine or an association module. For example, a model may be trained so that it learns by reacting to previous events. If the reaction with the user seems to indicate a successful outcome, a positive reinforcement cycle may occur, where the successful outcome promotes the strengthening of weights or paths in the system. The association between two variables, modules, or inputs and outputs indicates the relationship that forms between them, where the first change changes the later change, directly or indirectly. In some cases, multiple associations may be formed to or from the same module / variable or stimulus. This means that the next time the same event occurs, it will be prioritized as well. A second example may use negative examples, where weights are changed based on a negative response to some stimulus. Over the course of an interaction, this allows a prioritized set of weights to be built in the association engine, allowing the model's response to change and adapt. In some embodiments, the association between the stimulus and the weights may be indirect, filtered, or influenced by paths or other learning modules in the system. In some cases, particularly with a focus on biological systems, the learning process may preferably be indirect, where the full effect of changes is observable only in the behavior of the virtual object, i.e., it is not easy to see how a particular weight was changed to produce an effect.

[0188] In some embodiments, the learning module may depend on internal activity or internal sources or stimuli. This allows the system to generate an internal world rather than relying solely on external stimuli. For example, an internal homeostatic imbalance (e.g., fatigue) may affect behavior without being related to or associated with external input. Alternatively, behavior may be influenced by internally generated history / memory, thus resulting in an indirect internal response. In a further embodiment, a neural network similar to a weather system may constantly evolve its activity. This is because parts of the neural network exhibit chaotic behavior, depending on their past states in a nonlinear manner. In one embodiment, the system includes a neural network designed as a central pattern generator that generates consistent behavioral behavior even in isolation from external stimuli. For example, babbling may be generated by a rhythmically running central pattern generator (CPG), which may provide the basis for the learning module. In other embodiments, the learning network may form associations between internal stimuli, and the feedback loops have only indirect external outputs or are primarily focused on changes between interactions between modules.

[0189] It is not known that CPGs have been used for digital animation and provide a way to generate time series of activity (which may be periodic or rhythmic). This activity may be a core function of the system, is biologically plausible, and allows the neural network or computational elements of the module to generate patterns. For example, behaviors like crying or laughing or breathing are driven by a central pattern generator. The central pattern generator may be constructed or coded by fitting animations to a recurrent neural network (RNN) using a fitting algorithm. When the RNN activates (e.g., by an activation signal), a pattern may be generated. The generated pattern introduces variability into the animation, so that although the animation does not repeat exactly (because it is generated by or has a potential chaotic system as part of it), it has similar repetitions. For example, this may introduce variability in responses based on similar inputs, which we relate to some biological behavior.

[0190] In some embodiments, it may be desirable to fix the associations of modules. For example, a model may be trained to behave in a particular way, which may not need to be changed. Alternatively, the system's ability to change or adapt may change with time or neurobehavioral factors. In this case, the association engine or module may be referred to as a fixed module or a pattern module. A pattern module may be used for graphic elements that include pre-built appearances, including facial expressions. There may be a plasticity variable that can control the amount of change possible for the association module or the learning module. This may work by limiting the possible weight changes for any time step. The plasticity of the module may allow the rate of change to be adjusted, and may be optimized to trade off memory retention and new memory acquisition. In some cases, weights within a single module may have plasticity that changes. Plasticity may be influenced by feedback, as well as external regulatory signals, such as signals from neurotransmitters.

[0191] For example, the association module may be used to teach a module to play a simple game such as pong. The model may have a stimulus that is the visual input of a pong screen and may output the position of a paddle by moving its eyes. A means of reinforcement may be provided by an increase in neurotransmitters such as dopamine. When the model successfully hits the ball with the paddle through motor babbling, the model receives more dopamine and the neurobehavioral system associates this with ball / paddle tracking and strengthens the association module associated with this.

[0192] Illustrative Model Now referring to FIG. 6, the flexibility of the model is illustrated by the implementation of eye blinking. This function can be incorporated as part of the facial structure, such as the face 61, eyes 62 and cornea 63 described and shown in FIG. 5. The eye blinking is coordinated by the oculomotor system so that the blinking tends to occur during saccades. In one embodiment, the system can be modeled by a module with an appropriate neural network. In an animated model, the eye blinking system must also have graphic elements linked to the eyelids and the muscles around the eye. These connections are made simply thanks to the structured and combinatorial nature of the eye system and the computational data that is added. However, the eyes are also intricately related to other areas of the brain. For example, the rate of eye blinking may be influenced by dopaminergic activity. The structure of the model allows this completely different and complex system to provide input to the eye blink system without changing the eye system. If there is some form of feedback to the nervous system, this could also be simply coupled. The ability to couple the two systems to each other may depend on the hierarchical structure of the model as well as the individual nature of the nuclear modules.

[0193] In a further example, we might consider what happens when a change in the environment occurs. For example, when a change occurs in the visual field, this is detected by a camera. The output of the camera is mapped to a simulated retina that detects changes in luminance and maps them to activity in the superior colliculus. The superior colliculus resolves competing inputs and commands the oculomotor system (which includes multiple neural nuclei) to generate saccadic eye movements that are sent to the animation system. An unexpected stimulus triggers the release of dopamine via the tectonigral pathway. The release of novel or reward dopamine, which affects the basal ganglia, causes the eyes to foveate to the stimulus, modifying current motor activity and future responses by Hebbian plasticity. The amygdala may trigger the release of hormones in the hypothalamus and activation of motor circuits in the brainstem that activate the facial muscles, which associate the current emotional state with the stimulus and generate animation by activating precomputed biomechanically simulated deformations. The response and plasticity of the subsystems are influenced by the levels of various neuromodulators and hormones that also affect the emotional state. The animation arises from a complex nonlinear system dynamics that is self-regulated by physiological constraints of parameters, as the model's behavior is influenced by its history as well as external events. This depicts a specific set of responses based on visual environment changes as well as a biological understanding of the nervous system. However, it should be understood that complexity may not only manifest itself through the depicted model, and the model may be modified to incorporate or exclude sections or features as needed.

[0194] application To provide examples of possible uses of the described system and method, applications involving gaming and advertising environments are briefly described. However, the present invention is not limited to these embodiments. In gaming environments, users can control animated characters or virtual objects that interact with other in-game characters. In many games, the behavior of these characters is very simple or zombie-like. Using embodiments of the present invention, these in-game characters can be implemented, which helps overcome the computational or robotic nature or characteristics of many in-game characters. The computational nature can include immutable or unchanging interactions.

[0195] Characters based on neurobehavioral models can have a wide range of skills or can lack skills but express very realistic interactions. Furthermore, game developers can tune or train characters in simple ways to provide a wide variety of character types depending on the application. For example, characters can be modeled to be more or less sensitive to neurotransmitters such as oxytocin or dopamine or other variables (neuromodulation), or to have slower reaction times depending on the desired use of the character. The interconnected graph part of the model makes these changes clear and convincing. For example, an angry character, thanks to an interaction, can appear angry, and this angry facial expression can be quite realistic and / or smoothly changing based on the interaction between the user and the character. This can bring further advantages since it allows generating multiple characters with different interaction capabilities to generate complex worlds.

[0196] A character with a learning aspect or module would allow for the remembering of interactions between the character and the user, making the neurobehavioral model realistic if the characters meet again. For example, a character that the user treats kindly may have a learning module that remembers this and interacts positively in the next interaction. In this way, the interaction between the user and the character will be graphically realistic and develop in a realistic or interesting manner over a series of interactions.

[0197] In a further example, the system or method may be applied to or associated with an advertising vehicle, display screen or web page. In this example, the model may be interacting with a user to present an idea or product. The model may have a stimulus that is a visual input of the user's face and provide an animated visual output that the user can see. The user's interaction is interactive rather than one-sided, where the user's different responses to certain elements of information may cause the model to process the response and make a choice about what to present next or how to present this information. At the same time, the model's graphical or visual elements present realistic and emotive facial expressions to improve the relationship with the user. Because the model combines graphical and processing elements, a strong relationship can be built that at least partially overcomes the uncanny valley effect. Also, the model's neurobehavioral information may be used to determine the effectiveness of the advertisement. Again, the model's processing or graphics can be adapted to provide the desired service. For example, a drive-thru restaurant may have an order-taking model that has human-like characteristics to overcome any computer biases, but may be represented by a graphical output that resembles a character associated with the restaurant.

[0198] In a further example, the system may be used in a human-interacting environment. Airport check-in is often manpower intensive and requires high volumes of passenger processing. However, there has been limited success using avatars or digital entities for passengers. This is partly due to the lack of interactivity of the avatars. However, a neurobehavioral based avatar could potentially change behavior based on the passenger's response to provide better service. This may be achieved by visually monitoring the passenger or examining the answers provided by the passenger. The depicted model provides a system that is capable of sensing and changing behavior, as well as being deployable or adaptable to different situations or points of interaction with the passenger.

[0199] System Operation The system may run on a general purpose computer. The architecture of the system may include multiple levels or data structures. In one embodiment, there are multiple levels in the architecture of the model. The levels include: - Programming, which defines the type and function of each module; - Structures that combine and rearrange multiple modules; and A structure that provides links and communication between modules; may include: Dividing the model into a hierarchy containing modules at various levels of detail allows for a wide range of mutual connections between modules in the model. This is because while there is a clear separation between the modules, the overall structure provides a means to connect them. For example, if connections are made to the eyes, the eyes may be separate from the face or nose, allowing new connections to be made to the eyes without affecting the rest of the model. The organizational structure of the model also allows for the eyes to be easily found and links to be created. In some embodiments, connections may be made between virtually any of the variables in the model. This allows for interfacing graphics with the neural model as well as the creation of complex animated systems. In some embodiments, the described structure provides the ability to create complex animations in a simple manner to separate and differentiate design levels and skills.

[0200] This allows a first user to generate modules, a second user to structure and group the model appropriately, and a third user to form couplings and data flows between the modules. Coupling may be considered as a way of describing interrelationships between modules, or between variables of the modules or system. In some instances, one user may perform each of the tasks, but in a decoupled manner so that one section can be updated without affecting the system or requiring a large-scale rebuild or reorganization of the system. In this way, models may be built from libraries or collections of modules or module templates. This allows modeling to be separated into coding / preparation parts and assembly (of modules / module templates) and linking (of connectors) parts. The assembly part can be easily performed by someone familiar with the structure of the model, without necessarily requiring an understanding or skill set of the programming part. Module templates may be broad depictions of modules with no clearly defined functionality. These may be refined by the addition of functionality or graphics (e.g. from known scientific models) to form modules. Libraries may contain a mix of modules and module templates depending on the proposed construction method. The library may include a set of modules capable of generating a behavioral or neurobehavioral construct.

[0201] In one embodiment, an operator or user selects modules to include in the model or network, selects connections between modules, and adjusts parameters associated with the modules as well as the relative positions of the modules in the required structure. The operator then observes the digital entity or avatar displayed on the screen and adjusts a combination of module selection, position, parameters, and module connections that affect the characteristics of the avatar's response controlled by the model. For example, the characteristics of the avatar can be influenced or configured by changing the topology or layout of the required structure. New modules can be added, module sections are duplicated, or structural parts (e.g., tree branches) are eliminated. Changes in the connections between modules or changes in the system will also affect the characteristics (e.g., if neurotransmitters are no longer transmitted between modules or if the transmission is not effective). The amount of configuration required will depend on the level of differentiation between the behaviors of the avatars or virtual objects, or the importance of the differentiation.

[0202] In one embodiment, the additional information processing capability is based on the precise timing of various signals that are adjustable parameters to the system. Thus, the outcome of the model may depend on how the information is combined in the model. In one embodiment, the relative positions of the modules may be a parameter. This allows the movement of the modules to cause adjustments to the positional reference data. In one embodiment, a purely temporal reference frame is used instead of a positional reference frame.

[0203] Modules should be adapted to be recognized by the model initially or before use. A structure or hierarchy of multiple modules can then be generated. Each point or node on the structure may be linked to other nodes and may include links or pointers to modules from libraries. The nodes may occur with modules and may be referred to as modules. The computational elements or code of the modules may be directly contained within the modules of the nodes. After the required structure or dependency tree is built, connections can be made between the nodes on the structure. These connections may be considered as webs on the branches of the tree. The connections make it possible to transform the basic dependencies into complex evolving systems by linking points (using modules and connections) in a virtually unlimited manner. Connections can be made without the need for recompilation of the structure. In this way, the model may be updated, tested or optimized in real time and with fast response. The dependency tree may be referred to as a means of describing the system, and the relationships between the modules may be created by a programming language.

[0204] A feature of the model is that modules may represent the system at different levels of detail. For example, based on the hierarchical structure and design of the model, a simple basic system may be implemented to provide the basic functionality of the model. A part or branch of the structure associated with a feature of interest may be highly developed to a high level of detail (low level system) with many associated neurons or computational complexity, while the rest of the model operates at a relatively low level of detail (high level system). The level of detail required or used may be adapted depending on the situation or available processing power. This may be particularly relevant for models where the amount of detail required is difficult to predict. Whereas in previous models the level of detail had to be adjusted or selected before the construction of the model, the flexibility of the present system may allow for continuous manipulation of the level of detail.

[0205] Referring now to FIG. 5, a schematic diagram of a portion of the exemplary model 1 is shown. The schematic diagram illustrates the links, structures, and interactions between the model's graphical, computational, and biological properties. Starting at the bottom of the diagram, a vision module 50 is shown. The vision module may be implemented using a camera or other optical sensor or surrogate device. The module generates outputs including a pixel array 53 having a width 51 and height 52, and an arrangement number 54 (to distinguish between RGB and grayscale). These outputs may be coupled to other parts of the module. For example, they may be inputs to a face detection module 55. The face detection module 55 may be programmed or otherwise adapted to detect facial structures from an image and generate outputs describing a face 56, the face's coordinate location 57, 58, size 59, and a mask 60 of the image. Similarly, the output from the vision module 50 may be coupled to a motion detector 56 as shown, or to a range of other modules as needed.

[0206] A feature of the described system is that a module such as the vision module 50 may have any given structure within it. For example, a camera module may first be implemented as a black box for testing purposes, providing a known or constant input. If necessary, an appropriate imaging system or camera can be inserted into the module. This process requires that a wrapper or identification structure be organized, coded, or otherwise described around an input device (or model) to form a module. The definer of the wrapper or module tells the model how to interact with the module. Application of the wrapper may require specifying the module's inputs and outputs as well as time step behavior. After the module is prepared, introduction of the module into a model requires linking the module to a section of the model's hierarchy and linking the inputs and / or outputs (as defined by the wrapper) to other modules. The definition of the wrapper or model introduces or defines a new module, including different system or model types, into a system or model, such that the models remain distinct but are relevant and combinable. The characteristics of a module are not limited by the model in which it is placed. A model can incorporate multiple different modules with different kinetics or actions or neurons and combine them in a simple manner. The available modules may form a library or selection list where multiple pre-made modules may be incorporated multiple times into a model.

[0207] The type of module may refer to a particular module instance, but preferably refers to the central or overall functionality of the module. For example, the type of module may be graphical, where it has the primary or sole function of displaying input variables or displaying graphics using instructions of the input variables. Similarly, an animation module displays graphical elements, but also allows for the movement of the graphical elements. The type of module may be computational, where it has the primary or sole function of processing inputs into outputs. There may be various types or subtypes of modules, such as computational models. For example, a module may provide relatively simple computations, such as the "winner takes all module" of FIG. 5, or more complex models, such as neurobehavioral models or biomimicry models. These may be referred to as simple computational modules and scientific computational modules, respectively, but alternatively other terms may be used to distinguish between the subtypes of modules. For example, a neuron type module would include one or more neurons between the input and the output. An interface module may receive inputs from an external source or provide outputs to an external source. For example, a camera module would be designed as an input interface. A module may be a container or structural module that has limited or no functionality but creates structure or organization for other modules. A learning module may have memory or storage elements to provide changing behavior. The type of module may also refer to the level of detail of the module (e.g., high-level or low-level module).

[0208] The vision module 50, face detection 55 and motion detector 56 may be hierarchically organized under a visual system module 57. The visual system module 57 may be a container or may include or contain operational code or instructions. The visual system may also be linked to a processing module. FIG. 5 shows a neural network module acting as a salience map 58. The salience map module 58 may receive inputs from the face and motion detectors and generate a set of outputs related to important features of the visual image. This may be achieved by common known means. In some embodiments, for models based on biological systems, a biological type neural network may be used. The output of the salience map 58 may then be processed by, for example, a "winner takes all" module that isolates the strongest features. As shown in FIG. 5, the structure of the model allows all the building blocks of the visual system to be systematically contained or held together, arranged or related. However, the system also allows for coupling between modules, whether these modules are of the same structure, unrelated structures, or different container modules.

[0209] A visual system module 57 is shown in FIG. 5, which is included in the head module 58. The head module also includes a gaze system module 59, a brain module 60 and a face module 61. The possible module choices are not limited to those shown. Each module may include a part of the system required to build a model of the head. For example, the gaze system module 59 is in the head module and provides an indication of which direction the eyes are looking. This may require at least an input from the visual system 57, an output to the face system 61 and a signal to the brain module 60. However, further connections may be made to the brain or module 60, or to another module such as the eye 62 or pupil (not shown). The connections between modules (either at the container level or directly to a submodule) provide great flexibility in the running of the model. In a sense, any feature is represented as one or more modules whose inputs and / or outputs may be connected by connectors to any other inputs or outputs. The use of a modular system allows the layout or configuration of the components of the model, resulting in a visualization form of the model, as well as the configuration and / or reconfiguration of the model and its neural circuitry.

[0210] In one embodiment of the present invention, the graphical elements are integral to the model. This overcomes the problems associated with separately maintaining a complex biological model of the brain and the animations required for realistic movement. In the embodiment shown in FIG. 5, the face module 61 contains the graphics and animations for the modeled face. The face module and child modules such as eyes 62 and cornea 63 are included in the structure and hierarchy of modules alongside the processing and computational modules described above. In further embodiments, there may be further relationships between modules. For example, a connection may be made between the eye module 62 and the vision module 50 so that the camera looks in the direction of the eye. The combination of the graphical and computational elements is complex as each system is inherently complex with many inter-connections. However, the use of identical or closely corresponding structures for each system results in simplicity as well as understandable systems. This is due in part to the avoidance of redundancy between the various systems and the clarity of the resulting model to the user.

[0211] The hierarchical relationships of the graphic, sensor and processing / computational elements, as well as between them, result in a structure by which complex systems can be built and understood. Models can be built and improved piecewise without the need to carefully plan and optimize complex structures. For example, a modeler / user may start by defining the structure of a face and its components. After building the face 61, eyes 62 and cornea 63, appropriate graphics can be included within the hierarchy of modules. A simple feedback loop can then be programmed from the vision module 50 to instruct the eyes, possibly based on a biological model. Thanks to the described hierarchical structure, eye movements depend on head movements, leading to consistent and realistic movements. Effects can be passed through the hierarchy using deformation instructions contained in each module. Eye movements can also be linked to visual inputs that identify interesting environmental features. This input provides directional input to the eyes through links or connections. The eye system receives a combination of these inputs and operates to match the graphic and computational inputs. Novel features can be subsumed into the existing model by adding more modules and connections. This allows for the creation of modular skeletons and for increasing complexity of the graphical model to be added gracefully and without the need to rebuild the model from scratch.

[0212] In one embodiment, a model may be understood by considering the flow of information from inputs (e.g., external audio / visual inputs) to outputs. The necessary structure provides a first flow of information through the model. In particular, the necessary structure provides a layout or organization of modules such that information is shared among modules that depend on other modules. This organization provides a flow of information that is appropriate for understanding systems or physical structures where dependencies are clear and duplication and repetition of parts of the structure is desirable. The second flow of information is provided by the connections between the variables of the modules. This flow of information is constructed, preferably in a web-like structure on the necessary structure, to define links between modules that are not dependent on the necessary structure.

[0213] For example, these connections are particularly appropriate for neurotransmitters / neuromodulators or similar that have broad effects on multiple modules throughout the model. The connections or links also allow for modified or novel connections to be tested. Because they are, at least in part, independent of the required structures, considerable variation on the connections is possible, while the required structures ensure that the model remains consistent as a whole. The relative complexity of the required structures and connections can be varied. For example, in complex models, it may be desirable to use the required structures as a virtual framework with connections that allow links to be formed between them. However, for greater flexibility, simple required structures can be used with many more connections, and then information can be transmitted around the structure.

[0214] A necessary structure may indicate a clear relationship or dependency between two modules. For example, a necessary structure may represent the physical relationships of a set of modules (e.g., head, face, eyes, cornea, and pupil). In a model with limited necessary structure, this information may be transmitted by couplings, but this makes it more difficult and less efficient to visually represent parts of the model, as well as for users to change the configuration of modules. Couplings require more data to be transmitted between modules, and couplings need to be carefully constructed to ensure that important or necessary relationships are not eliminated. Couplings then link variables between modules without establishing dependencies between the modules themselves.

[0215] In some embodiments, the required structures may form a significant part of the operation of the system. Preferably, the operation is time-stepped. The time steps should occur at a rate fast enough that the animation or output appears fluid. A frame rate of 40 frames per second may be perceived as fluid, requiring a new frame every 25 ms. Given a computation time of 1.5-2.5 ms, this allows for approximately 10 computational time steps before a redraw is required. However, the graphical output may be redrawn more or less frequently as required, and the computation time may vary depending on the system parameters and the complexity of the model. In one embodiment, the main operation may involve each module taking a time step. At this time step, the computational elements of the module may be updated. Typically, the graphical elements do not have any changes for the time step. Each connector is then updated with new variables and transmits these to those variables for the coupled modules. The model can then be redrawn according to the transformation instructions, preferably by working from the top to the bottom of the required structures. As mentioned above, timestep and connector updates can be repeated multiple times before redrawing. Similarly, some modules may have breaks or holds so that they do not update as frequently as the rest of the model.

[0216] A model may be implemented by having a first library or collection, or a list containing a very wide range of template modules or module types, such as neuron modules, association modules, etc. This library may be used to build a set of modules implementing or describing a particular method or model within a second library or module template. The modules are then described, preferably in BL, with a description of the links between each of the (as modules can preferably be used multiple times in one model) or a description of each instantiation of the. This description explains how the modules are combined and which modules have a graphical output. Further descriptions or code may be used to provide further structure or links between the modules. In a preferred embodiment, at least one of the descriptions is based on a file structure, where the modules may be arranged in a file structure to establish the arrangement, required structure or hierarchy. Such a system provides a high degree of flexibility in modifying the model, separating the complexity of generating one model from the complexity of appropriately combining a range of modules.

[0217] Neurobehavioral models of the cortex Referring now to FIG. 9, there is shown a schematic diagram of the cortex 20 and a selection of connections to and between associated modules. This may be referred to as the cortical-thalamic-basal ganglia-loop. The cortical modules may have neuronal module(s) 23 and / or synaptic weight modules 24 that integrate the activity of incoming modules, or association modules that may be plastic or change their effect over time. Input to the cortex 20 comes from a sensory map 21. The sensory map may be used to process data received from an external stimulus, such as a camera 17. The sensory map 21 serves as a pixel-to-neuron translation of stimuli that may be input to the cortex.

[0218] The cortex may have feedback connections 33 with other modules such as the thalamus 22. Feedback loops can be used to provide a means of integrating sensory perceptions to the cortex. Positive feedback loops may aid in associating visual events or stimuli with actions. The cortex is also connected to the basal ganglia. The basal ganglia 29 may have multiple associated modules or submodules, including neuronal and synaptic weight modules, that provide feedback to the cortex or via the thalamus to the cortex. Although only a single connection 31, 32 is shown to the basal ganglia, multiple connections may be formed and further connections may link to modules shown or not shown. The basal ganglia 29 itself, or modules connected to the pathway, may modulate the feedback between the cortex and the thalamus. That is, intermediate modules with neural functions may increase the complexity or adaptability of the structure. Neurotransmitters / neuromodulators 25, such as dopamine or oxytocin, may be used to affect the operation of the structure. This may be implemented as a connection from another part of the module or from an external stimulus. In a preferred embodiment, neurotransmitters such as dopamine are linked from reward values ​​26, where high reward values ​​are positive and will cause more dopamine to be released.

[0219] The cortex may also be linked to output means, which in the illustrated diagram is a motor output means for muscle activations 28 coupled via a brainstem module 27. The brainstem may include a pattern generator or a recurrent neural network module with pre-defined or fusion-able pre-defined muscle activations 28. Output means may also be used to display the operation of any of the modules in the model. These are separate from the animation or muscle activation display, allowing the display of changes in the variables, synaptic weights or other features of the model. As explained herein, this may be achieved by each module having a functional graphic component, which can be switched between visible and invisible as required. Alternatively, the graphic module may be used to improve the display or calculations associated with these outputs. For example, a scrolling plot 30 may be linked to the basal ganglia 29 to monitor the changing nature of any one or more of the variables. Since another module is used, more calculations regarding the variables plotted, or the display of the plot, may be possible. In another graphical output, not shown, activation or changes in synaptic weights may be modelled, making it possible to visualise transmission between neurons between modules or within modules.

[0220] Eye movement model Figure 10 shows a system for controlling eye movements of an avatar displayed on a screen, receiving stimuli for the avatar with a camera. The camera communicates with a computer vision library that communicates with a face detection unit. The network has the following modules: SC: superior colliculus; Tr: trigger neurons; EBN: excitatory burst neurons; LLBN: long lead burst neurons; OPN: omnipause neurons; MN: eye movement neurons; Physics; physics-based dynamic inhibition. The modules interact to form a biological type system based on the connections between them.

[0221] FIG. 11 shows an expanded system or model with an exemplary emotional response module that responds to the presence of a face (FFA) and the model's corticotropin releasing factor (CRH), B-endorphin (BE) and Oxytocin PHYSIC (OXY) levels. The OXY parameters are neurotransmitters that can change or affect the performance or operation of a module, such as CRH. For example, higher OXY values ​​allow for greater inhibition of CRH, which reduces stress and the likelihood of triggering activation of the pain behavior circuit that activates the PL. Additional modules are face recognition (face recognition module); CRH (corticotropin releasing hormone); BE (beta endorphin); OXY (oxytocin); FFA (fusiform face area); ZM (zygomatic major); and PL (platysma).

[0222] Adjustment of a parameter (e.g., the level of a neurotransmitter such as oxytocin) is shown in FIG. 12. The parameter may define an aspect of the functional operation of the module. In this case, the response of the digital entity or avatar and / or the characteristic response of the digital entity is adjusted by adjusting the parameter associated with the functional module. In this way, the characteristic may be changed in a global sense, rather than requiring a series of changes in each module. For example, in some embodiments, the adjustment may propagate radially from the module, or as a point in the space in which the network is defined, or linearly across the network, or along the connections, or through the necessary structure. In the case of the network of FIG. 12, the OXY parameter drops by 50%. If this reflects the biological oxytocin system, the dynamics of the system are altered, the system becomes more susceptible to stress, and the delay in activation of the pain circuit that activates the platysma muscle is reduced.

[0223] FIG. 13 includes additional external inputs or stimuli provided by microphones and speech recognition modules for face and speech multimodal recognition. When face and speech recognition are simultaneous, the multimodal recognizer R is excited. The time at which both signals arrive at R may depend on the required structure or connections or different processing paths. In this example, a 50 ms time delay in the module or connection ensures proper signaling time. In alternative embodiments, delays are used to ensure that signals arrive at the module at the same time or at the proper time step. FIG. 14 adjusts the delay in the speech recognition connection, which affects the relative timing of the arrival of the speech and face recognition signals. In this example, an additional delay is added compared to the network of FIG. 13.

[0224] Development of eye movement models FIG. 15 shows a schematic diagram of a model that receives visual / auditory input and generates output animation. The embodiment of FIG. 15 shows that complex biologically based architectures can be built with high levels of model refinement and the ability to further increase functionality as needed or required. Part of the required structure is shown where modules are formed, which may be modular in some embodiments. In a complete model, the required structure may be larger, fitting each module into a hierarchy or other structure. The structure of the model allows for the incorporation of inter-coupling between modules after their construction, resulting in further complexity as the model develops. Although this example describes a face recognition system, the model is not limited thereto and the face recognition system may be only a part of a complete model.

[0225] We first consider an initial system that includes only a computer vision module, an eye movement module, a physics module, a facial animation render module, and a screen module. Additional parts of the model may be built from this foundation. Computer vision input, for example from a camera, is fed into a computer vision library that is used to detect faces. The computer vision input also shows how "black box" functionality or strictly computational elements can be integrated into a biologically based system. Face detection, or a similar restriction or concentration of the input field (vision in this case), reduces the complexity of the input data to the model. This generates a target that is sent to the superior colliculus (SC) module, which causes a saccade (fast eye movement) that produces activity in motor neurons that cause eye acceleration.

[0226] The physics system dampens the movements due to inertial constraints, and the actual eye movements are fed back to the SC for correction. The physics module helps to reduce the likelihood of unnatural movements, for example by limiting the speed of response, as well as to enforce the physical constraints. The calculated eye movements are fed to a facial animation system that rotates the face geometry appropriately and renders it on the screen. The eye movements can be fed back to the computer vision system to generate the foveal region or gaze direction. This allows the input to be related to the output of the model, creating a feedback loop or dependency within the model.

[0227] More complex models may also include facial expression detection and expressive responses. For example, visual cortex, limbic system and brainstem PG may be added to the model. The fusiform face area may be a convolutional neural network (CNN) for the recognition of facial expressions that trigger different emotional behaviors (e.g., fear via platysma, smiling via zygomatic major). A central pattern generator (CPG) may be used as the basis for the actions required in response to emotions. The facial nucleus breaks down the activity of the facial muscles and sends animation weights to a facial animation system that deforms the geometry, which is fed to a face renderer and then to the screen.

[0228] Neurotransmitters / neuromodulators may also be incorporated into the system through their connections to the amygdala (AMG). The amygdala has connections to the autonomic nervous system (e.g., via the hypothalamus) and endocrine systems (where these may be related to biological projections). Oxytocin (OXY), corticotropin releasing hormone (CRH) and beta endorphin (BE) levels have a mutual regulatory effect and are used in this example to regulate the triggering of the facial circuit in the brainstem. The brainstem CPG generates patterns that control the facial muscles over time. The ventral tegmental area (VTA), which produces dopamine, has anatomical connections from the SC and AMG, providing an example of separate neural systems that are interconnected. The ability to combine modules separately from their organization allows additions to the model as well as modifications to the model in a simple manner.

[0229] Additional inputs may be included in the system (e.g., an audio processing system), which may detect speech. The incorporation of new modules may also require new modules in other module blocks (such as the extension of the cortex to include a multimodal integration component to fuse or combine auditory and visual inputs). However, the addition of new modules does not necessarily require modification of previous connections, simplifying the extension of the model. A further addition to the model may be achieved by the synchronization of eye blinks between the oculomotor system and the facial nucleus. Eye blinks (involving the palpebral part of the orbicularis oculi (OOc)) are synchronized by saccades.

[0230] A blink neuron module is added to control the OOc muscle and the timing is synchronized through a connection with the oculomotor system. A second step may introduce a connection from the blink neuron to the dopamine system. The spontaneous eye blink rate (EBR) has been shown to be a clinical marker of dopaminergic function. A modulatory dopaminergic connection is formed from the VTA to the blink neuron. The dotted connection shows how dopamine can modulate the blink rate by adding a novel connection between separate systems. This modulates the blink rate but is still synchronized with saccadic activity, showing the flexibility of using the model to couple different neural subsystems together. The modules form the necessary structure, for example including a set of modules or associated modules, but can be linked by connections within or outside these groups. These connections allow the incorporation of high-level effects or the synchronization of effects across different module groups.

[0231] Further structure Referring again to FIG. 5, a system is shown having a hierarchical structure with modules that are often contained within other modules. In one embodiment of the system, individual modules may be referenced by referencing down the tree from the top module to the base module. For example, an FFA shader may be referenced as head / brain / FFA / shader. Alternatively, combining inputs and outputs to modules higher in the hierarchy may be preferred in some embodiments. For example, a commonly used output such as face 56 may be assigned as an available output to the visual system module to make face reference easier.

[0232] Referring now to FIG. 7, a depiction of a module 100 is shown. The module 100 includes data elements 101 related to computation 103, data elements 102 related to graphics 104, and elements 107 related to the transformation of the module. Any subordinate parts of the module may be included in the module 100, but preferably as a hierarchical tree structure in which the modules are included. The tree structure has a central point or central module from which multiple modules branch, and each lower layer of modules can have further child modules. In some cases, the tree structure may have additional modules outside of the main branch. The inputs 109 and outputs 108 of the module 100 may be elements or variables involved in any one or more of the module's dependencies. Graphic elements or data 102, which include a set of modes 105 associated with the operation of the module, and a set of shaders 106 that generate appropriate levels of light in an image. Alternatively, the module may provide a graphical output that visualizes part of the computation. A computational element may contain instructions or pointers to computational blocks contained in a library structure or the like. In some cases, a computational element may be bounded and the module acts as a constant or container module to improve the hierarchical dependency structure. In other embodiments, a computational element may include a large and complex neural network or a single neuron.

[0233] Transformation elements may provide data that regulates how the graphics of a module can change as part of an animation, or how changes to subordinate structures affect the graphical element. This is particularly important when a hierarchical structure is used to traverse the model. Each module may have a transformation section that provides instructions on how to react to changes in modules above it in the hierarchy. For example, when a face turns, the facial features should rotate as well as features contained in the brain. Rotating the face will affect the rotation of the eyes, which may affect the appropriate rotation of the pupils. The hierarchical structure provides a means to make these changes consistent. So, when depicting an element, the changes in the element can be appropriately combined with the surrounding elements to generate realistic animations. Although the description of transformations is based on the hierarchical structure of the model, it should be understood that alternative structural methods of linking the transformation means in different ways with similar results may be used.

[0234] architecture The system structure may include a first and a second model substructure (data structure), where the first substructure (level) is defined by the arrangement of multiple computational modules and the second substructure (level) is defined by connectors linking the variables of the modules. The first substructure may be a scene graph, which is directed and graphical. This may allow for careful arrangement of the modules. The second substructure may be a directed graph, where the connections form the edges and the modules form the vertices or nodes. These two levels of substructure increase the effectiveness of the running of the model, since the data is separated from the control code. Thus, the modeling process is not a matter of building a fully linked system or designing a process flow, but rather a way to link multiple modules from the first substructure using the second substructure (which may be by using the variables of the modules). The structure also allows for variables or constants to be updated while the model is running, since the model does not need to be recompiled if the relationships or connections are separate from the data.

[0235] The first sub-structure may be implemented as a number of modules or as a structure into which the modules are organized. The second sub-structure may be implemented as a set of instructions for combining the modules. In some embodiments, the set of instructions may be located in a number of separate files. Each separate file may define a portion or subsection of the combination of the model. In certain embodiments, the instructions may be located in the same structure as the modules, but separate from the modules.

[0236] First sub-structure (module) The first level may contain an organized structure of modules. In some embodiments, this may be a tree-type structure in which the modules are substantially hierarchically organized. The modules may be arranged in a directory-like folder structure. This is particularly useful when container modules are present. FIG. 3 shows a possible structure in which a container module "Scene" holds several modules, including a container module "Face". The module "Face" holds two further modules "Eyes" and "Mouth". This may be stored in a file-like structure in which "Scene" is the top folder, "Face" and "Head" are first level subfolders, "Eyes" and "Mouth" are subfolders, and so on. In this way, the model structure is clear and easily viewable. Module elements may be copied or duplicated by copying the required folder levels and all contained folders. This may be useful, for example, when the eyes need to be independent of each other. The same model structure is duplicated, but the eyes may each have different control signals or minor changes may be made.

[0237] Second Substructure (Connector) A sub-structure contains a set of instructions related to a module. The instructions may be contained in a single file related to an entire model or an entire animation. In a preferred embodiment, a second sub-structure contains a set of separate linked files. In one embodiment, the instruction files are contained in the same structure as the modules. They are contained one (or more) hierarchical level above all models that are subordinate to them in the required structure. For example, instructions to link "eye" modules may preferably be in a folder containing a "face" module. However, the instructions may also be located in a folder containing a "scene" module, or at any level above the "face" module.

[0238] It is advantageous to place instructions at the level immediately above the modules they reference, because this results in an efficient modeling technique. In particular, if a change needs to be made to a module or to its instructions, the correct location can be easily found. The collocation of modules and associated instructions then allows the entire module to be quickly duplicated with the appropriate instructions. This can be useful for moving a module to a different model or for copying a module internally within a model. In one embodiment, there is a separate instruction for each stage of the first substructure, so that The "direction" command is in the "eyes" folder. The commands for the "eyes" are in the "face" folder, -The commands for the "face" are in the "Scene" folder.

[0239] Operation When the model runs, it compiles a first and a second sub-structure (preferably arranged in a directory tree as described above) that contains configuration files to generate its modules, connectors, geometry, etc. The required structure may be in the form of a directory tree where multiple modules and links between them can be built, which may be changed within the structure. At each time step, the structure needs to be traversed and updated. This may proceed in either a bottom-up or top-down manner depending on the particular design of the model as described above, but is preferably top-down in a hierarchical structure with the head at the top. Each module is evaluated based on the inputs it is providing. This includes all container modules and their child modules. If a module does not have code, such as a container module, no changes will occur. However, if code or computational data is present, these will be executed, and these are usually independent of any other part of the system. The results of the time step are then transmitted to the output fields. In a second pass through the structure, the outputs may then be replicated across the connections. This updates each of the module's inputs for the next time step. In some cases, there may be thresholds that may be updated in one or both of the processes or stages performed on the joins (e.g., holds). If a substantial change occurs or a set time has passed, the model may be completely rebuilt, including substantially all elements, to ensure continuity.

[0240] In the specific example shown in FIG. 4, files or folders with the same name (without any file extension) shall contain data that belong to the same object. For example, geometry data (.obj, .frag, .vert, .geom, .mtl, .tex, .trans, or image files) should have the same name as the object it belongs to and should be placed inside a folder with the same name. Object definition files or instruction files (.blm, .blc) may be placed in the same parent directory as this folder. As shown in FIG. 4, a simple module can be specified that has geometry, shaders, but no textures. Thus, when the code is running, it reads and combines items with a common name, which bring further detail to the model.

[0241] Module If new modules are required, they can be created separately from the modeling system. Modules may vary depending on the embodiment and the particular model, but may include: ·animation Provide a known time step Includes animation files Folder module Also known as the Container Module · Holds other modules Neuron Module For example, leaky integrate and fire module Multiple leaky integrating neurons ·Synaptic weight module · Potentially combined with neuron modules to form self-contained artificial neural networks Visual interface module Scrolling display module for showing output Interface module Vision module Possibly control interaction with the external world (e.g. camera or microphone) Constant value - Time step independent Black Box A proxy module that performs a task or is updated later. Nothing Empty modules can be ignored. Additional modules or types of modules may be created as needed.

[0242] Before a module can be used in the modeling environment, it must first be created. This involves defining the model's inputs and outputs and the relationships between them. The module definition is then deployed.

[0243] For example, consider the well-known model of a neuron (the leaky integrate-and-fire neuron), which

number

number

[0244] A module definition lists its variables and describes the behavior of the module when used. Variables are important because they allow the formation of bonds or links between modules. Variables are a way to access the parameters of a module in order to form links or bonds. Some variables may refer to multiple values, while some may only refer to one value. Some variables are designated as output variables. These variables are the output of the module's computation and are defined such that they may not be changed from the outside, effectively "read-only" variables. Other variables are designated as input variables. These variables affect the module's computation of its output, but are not themselves changed by the module. They are "read-write" variables that can be changed from the outside, or simply read. A variable may be designated both an input and an output. This means that it may be changed from the outside, but its value may also be changed as a result of the module's computation.

[0245] All module types, except for container modules, may require a file to configure its type and its parameters. The first line of this file typically identifies the type of module it is creating. Subsequent lines contain settings relevant to the type specified in the first line. For a particular model or animation, the module file (e.g., .blm) may generate an object based on one of the defined modules, such as a leaky integrate-and-fire neuron. A module is inserted into a model by defining the required inputs and outputs. Code section A shows example code for a module. Ignoring formatting, this example module first names the module type, then lists each of the inputs followed by a default value. If an input is not indicated when inserting the module, the default value may be used. In this way, a neuron model needs to be created only once and may then be inserted into an animation or model at multiple points by creating the appropriate .blm file.

number

[0246] Variables and Bindings Variables and connectors provide links between modules of the first substructure. Variables provide the means by which parameters of a module are accessed by a connector. Some variables may refer to multiple values, while some may refer to only one value. Variables may be defined to be available and editable internally or externally as needed. Modules may have several variables, which may be either input or output (sometimes both). Output variables are determined by the computation process of the module that owns them and may be read but not modified by connectors. Input variables are input parameters to the computation of a module and may be both read and written by connectors. When referring to variables (or any module or connector for that matter), the syntax reflects a hierarchical directory-based data structure.

[0247] Variables may be created as part of the definition process of building a module. Variables can be linked together by instructions to generate a model or animation. Instructions link one or more variables so that at a time step the variables may be passed between modules. In some embodiments, variables may also have holds, pauses or other processing as they are passed between modules or may be otherwise timed. In some embodiments, variables may have sub-variable elements, which provide a means to reference groups of variables or elements of a group. For example, a file named Texture Data may have the following three sub-variables: texture.data - texture color data string reference texture.width - reference to the texture width (texels) texture.height - reference to the texture height (texels)

[0248] To reference a module, a directory path to the module may be used, starting with the directory in which the reference is made. For example, if the module "test_module" is located in the same directory as the connector, the module is simply called "test_module". However, if test_module is a child of a module called "parent_module" and the connector is in the same directory as parent_module, then "parent_module / test_module" is used. Variables may be considered children of their parent modules and are referenced using the same hierarchical syntax. If test_module has a variable called "output", this variable is called "test_module / output". The same rules for directory paths above may apply. To reference a variable output when in the parent_module directory (see previous paragraph), one needs to use the path "parent_module / test_module / output". In such an embodiment, it may be observed that instructions contained in files near the modules associated with them provide simple names.

[0249] connector The second sub-structure links or couples modules together. Connectors link the variables of modules together. A file that defines a connector may contain an identifier, preferably in the first line, indicating the type of connector to be created. A connector's file also contains type-specific information. Generally, however, there will be at least one line where an input variable is transmitted to one or more other variables.

[0250] Possible connector types include, but are not limited to: Identity Connector · Strict equality Simple and common

number

[0251] In the above description, reference has been made to specific components or entireties of the invention that have known equivalents, and therefore such equivalents are incorporated herein as if individually set forth.

[0252] Unless the context clearly requires otherwise, throughout the description and claims, the words "comprise," "including," and the like, are to be construed in an inclusive sense, i.e., "including but not limited to," as opposed to an exclusive or exhaustive sense.

[0253] While the present invention has been described by way of example and with reference to possible embodiments thereof, it is to be understood that changes or modifications may be made thereto without departing from the scope or spirit of the invention as defined in the appended claims. The above-described embodiment can also be described as follows, but is not limited to the following. [Configuration 1] 1. A computer-implemented system for generating interactive behavior of an embodied virtual character or digital entity, comprising: The system includes a plurality of interconnected modules that collectively represent a neurobehavioral model; At least one of the plurality of interconnected modules receives data characterizing an external stimulus input from a real world; the interconnected modules process the data in a time-step manner and provide an external output defining a response of the virtual character or digital entity to the external stimuli; A system in which time steps in the modules occur based on the number of computational time steps provided before each external output. [Configuration 2] 2. The system of claim 1, wherein each module has at least one variable and is associated with at least one connector, the connectors communicating information regarding variable values ​​between modules using a standardized time-based network format. [Configuration 3] 2. The system of configuration 1, wherein the external output is at least one of a graphical output, an audio output, and the actuation of a physical robot. [Configuration 4] The system of configuration 1, wherein the external stimuli input from the real world are received from one or more of the group consisting of a camera, an electromagnetic transducer, an audio transducer, a keyboard, and a graphical user interface. [Configuration 5] 2. The system of claim 1, wherein the interconnected modules are configured to exhibit chaotic behavior in a nonlinear dependence on past states characterized by the variable values ​​of the modules. [Configuration 6] The system of configuration 1, wherein at least one of the plurality of interconnected modules receives data from another one of the plurality of interconnected modules characterizing an internal stimulus, the internal stimulus influencing the external output that defines a response of the virtual character or digital entity to the external stimulus. [Configuration 7] The internal stimulus is The internal homeostatic processes modeled in said neurobehavioral model; Associative learning modeled in said neurobehavioral model; and Central Pattern Generator The system of configuration 6, generated by a subset of modules that mimics at least one of the following: [Configuration 8] The system of any one of configurations 1 to 7, wherein the plurality of modules have coupled computational and graphical elements, each module representing a biological process, causing the computational elements to relate to and simulate the biological process, and causing the graphical elements to visualize the biological process. [Configuration 9] The system of any one of configurations 1 to 7, wherein the plurality of interconnected modules that collectively represent the neurobehavioral model simulate a plurality of parallel systems for generating behavior, and at least one module is configured to resolve conflicting inputs from a plurality of other inputs from the interconnected modules and provide the external output. [Configuration 10] 10. A method of controlling a display generated by a computer using a network of interconnected modules according to any one of claims 1 to 9, having defined functionality, wherein the modules communicate using a format for time-varying data, and the couplings introduce timing and / or delay attributes into the time-varying data depending on the placement of the coupled modules in the network, and wherein the response caused by the network can be adjusted by rearranging the modules or the couplings.

Claims

1. 1. A computer-implemented system for generating interactive behavior of an embodied virtual character or digital entity, comprising: a plurality of interconnected modules collectively representing a neurobehavioral model, the neurobehavioral model being a model based on a biological nervous system, at least one of the plurality of interconnected modules receiving data characterizing an input stimulus from a real world; the neurobehavioral model is configured such that the plurality of interconnected modules operate in a time-step manner to provide an external output defining a response of the virtual character or digital entity to an external stimulus; 1. The system of claim 1, wherein at least one of the plurality of interconnected modules has a muscle-level component for facial animation, the muscle-level component being driven by the neurobehavioral model and having muscle-related graphical elements and computational elements related to muscle actuation, each graphical element having an individual graphical output, and wherein outputs of a collection of similar muscle-level components are combined to form a cohesive output effect for a larger muscle or muscle region.

2. At least one of the plurality of interconnected modules receives data characterizing an internal stimulus from another one of the plurality of interconnected modules; The system of claim 1 , wherein the internal stimulus is associated with the external output that defines a response of the virtual character or digital entity to the external stimulus.

3. The internal stimulus is The internal homeostatic processes modeled in said neurobehavioral model; Associative learning modeled in said neurobehavioral model; and 3. The system of claim 2, wherein the pattern generator is generated by a module that mimics at least one of: a central pattern generator.

4. The system of any one of claims 1 to 3, wherein the real-world stimuli are received from one or more of the group consisting of a camera, an electromagnetic transducer, an audio transducer, a keyboard, and a graphical user interface.

5. The system according to any one of claims 1 to 4, wherein the plurality of interconnected modules are configured to exhibit chaotic behavior in nonlinear dependence on past states characterized by the variable values ​​of the modules.

6. The system of any one of claims 1 to 5, wherein the external output is at least one of a graphic output and a robot operation.

7. The system of claim 1 , wherein at least one of the plurality of interconnected modules generates an association between the external stimulus and the external output.

8. The system of any one of claims 1 to 6, wherein at least one of the plurality of interconnected modules has an adjustment value that adjusts a function of one of the plurality of interconnected modules.

9. The system of any one of claims 1 to 8, wherein the external output is any one of a change in orientation, a change in emotional expression, a change in body language and mirroring of an external input.

10. The system of any one of claims 1 to 9, wherein the plurality of interconnected modules represent a plurality of parallel systems operating at time steps to define a response of a virtual character or digital entity to the external stimuli.

11. 1. A computer-implemented system for generating interactive behavior of an embodied virtual character or digital entity, comprising: a plurality of interconnected modules collectively representing a neurobehavioral model, the neurobehavioral model being a model based on a biological nervous system, at least one of the plurality of interconnected modules receiving data characterizing input stimuli from a real world, the plurality of interconnected modules being arranged in at least one circuit; the neurobehavioral model provides external outputs that define responses of the virtual character or digital entity to external stimuli; At least one of the plurality of interconnected modules has a muscle-level component for facial animation, the muscle-level component driven by the neurobehavioral model and having a plurality of muscle-related graphical elements and computational elements related to muscle actuation, each graphical element having an individual graphical output, and the output of a collection of similar muscle-level components is combined to form a cohesive output effect for a larger muscle or muscle region.

12. The system of claim 11 , wherein the at least one circuit replicates a neural circuit.

13. The system of claim 11 or 12, wherein at least one of the plurality of interconnected modules represents a plurality of functionally distinct parallel systems to define a response of a virtual character or digital entity to the external stimuli.

14. 1. A computer-implemented system for animating an embodied virtual character or digital entity, comprising: a plurality of modules, each of which may have a computational element, a graphical element, and one or more variables; At least one of the plurality of modules has a muscle-level component for facial animation, the muscle-level component being driven by a neurobehavioral model, which is a model based on the biological nervous system, and has a number of graphic elements related to muscles and computational elements related to muscle actuation, each graphic element having an individual graphic output function, and the output of a collection of similar muscle-level components is combined to form a cohesive output effect for a larger muscle or muscle region; At least one of the plurality of modules is adapted to modify an appearance of the graphic output function; At least one of the plurality of modules is an association module having weights linking input variables and output variables.

15. The system of claim 14 , wherein the association module is a learning module.

16. The system of claim 14 , wherein the association module comprises a plasticity variable configured to control an amount of change allowed to the weights linking input and output variables.

17. The system of claim 16 , wherein the plasticity variable is influenced by feedback.

18. The system of claim 16 , wherein the plasticity variable is influenced by an external regulatory signal.

Images