Device, process and computer program for simulating physiological behavior of mammal using virtual mammal

Abstract

To provide a device, process and program for simulating physiological behavior of a mammal using a virtual mammal.SOLUTION: The device including a mobile head and mobile eyes comprises: a visual stream for successive data representative of eye poses; instructions for mobility action instructions; a memory for storing information on an environment and on stabilization constraints; and a processor assessing and recording a current part of the environment, triggering successive movements of the head and of the eyes according to a function of the mobility action, the successive data and the information on the environment, and controlling the successive movements of the eyes with respect to the successive movements of the head by using the stabilization constraints according to a function of the successive data. The device further comprises an input for training movement sequence data representative of a training movement sequence of the head and the eyes associated with a training environment. The processor learns the stabilization constraints by triggering, in the training environment, the successive movements of the head corresponding to the training movement sequence, and assessing and recording a current part of the training environment from the sequence data.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The invention relates to a device for simulating the physiological behavior of a mammal, more precisely the behavior related to the visual system of a mammal placed in an environment, by means of a virtual mammal. The invention can be implemented in the computing field and has many applications, for example in the ophthalmology field for the evaluation of vision correction or the testing of ophthalmic products. [Background technology]

[0002] Much research has been done on the visual system of living organisms, particularly mammals, and generally humans among mammals, from the structure and operation of the eye, light transduction, to the neural structure and operation related to vision. Based on such knowledge and the behavior of the visual system, simulation models have been realized in computers. Tools (e.g. CarlSim), application blocks (e.g. software body avatars from www.makehumancommunity.org), or global application systems by combining blocks have been proposed to simulate the behavior of living organisms in the field of robotics, but this is generally quite limited, especially with regard to the structures to be simulated and the environment in which they are created.

[0003] Moreover, in humans, the visual system is generally evaluated under very constrained conditions with respect to the images provided to the visual system, in particular with fixed images, with no head and eye movement, which can be referred to as stationary or quasi-stationary conditions.

[0004] Here, visual systems are exposed to extremely changing visual environments in reality, and any effective evaluation of visual performance should take into account such changing environments. For example, document (Patent Document 1) describes a method for objectively or subjectively determining or selecting visual equipment, particularly lenses, where the visual equipment, the visual environment, and the wearer can be virtually simulated. The importance of taking into account the motility of vision, particularly with changing visual environments, and the mobility of the head and eyes can be further emphasized with respect to the fact that such changes and mobility are essential during the development of the visual system of an organism to generate a correctly functioning visual system.

[0005] Moreover, with regard to eye movements, two main causes can be considered: reflexive movements and other movements as opposed to reflexive movements. These other movements can be termed spontaneous movements as opposed to reflexive movements. In particular, when the head is moving, the visual system responds to the detection by the vestibular system of head movements with a reflex response that also depends on the currently visualized environment. This is called the vestibulo-ocular reflex, VOR, and allows the stabilization of the image on the retina during head rotation by contralateral eye movements that keep the image in the center of the visual field. The VOR is mediated by the brain region where adaptation is directly driven by sensorimotor errors, namely the cerebellum. The cerebellar adaptation process, driven by an error signal in fixation signaled by retinal slip, minimizes the retinal slip that cuts the vestibular output to generate a compensatory motor command that ultimately drives the eye movement. The vestibulo-ocular reflex is operable whatever the visual conditions are, and functions in both light and dark conditions. The vestibulo-ocular reflex has been the subject of the following conference and related papers: (Non-Patent Document 1)

[0006] Such advanced research into vision systems has in any case proven to be complex to integrate satisfactorily into the realistic implementation of a robot or digital avatar moving in a given environment: existing simulation systems usually either fall short of a realistically sufficient vision mechanism or prove to produce more or less substantial distortions for real-world mammals.

[0007] The document (Patent Document 2) is also known. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] European Patent Application Publication No. 1862110 [Patent Document 2] International Publication No. 2018 / 005891 [Non-patent literature]

[0009] [Non-Patent Document 1] "Exploring vestibulo-ocular adaptation in a closed-loop neuro-robotic experiment using STDP.A simulation study" Francisco Naveros et al., 2018 IEEE / RSJ International Conf. on Intelligent Robots and Systems (IROS), 10.1109 / IROS.2018.8594019. Summary of the Invention [Means for solving the problem]

[0010] One object of the present invention is to provide an apparatus for simulating the physiological behavior of a mammal in an environment by means of a virtual mammal, said virtual mammal comprising: a head that moves relative to the environment; at least one eye that moves in rotation relative to the head, The apparatus comprises: at least one first input adapted to receive continuous data representative of a pose of said at least one eye with respect to an environment; at least one second input adapted to receive instructions for at least one movement behavior by the virtual mammal in the environment; at least one memory adapted to store information about an environment and stabilization constraints between the at least one eye movement and the head movement; at least one processor, assessing a current portion of the environment from said continuous data and, if said current portion is not identified in said at least one memory, recording information regarding said current portion in said at least one memory; triggering continuous movements of a head and of the at least one eye in function of the at least one movement behavior, the continuous data, and stored information about an environment; at least one processor configured to control dynamic adjustment of the continuous movement of the at least one eye to the continuous movement of the head in function of the continuous data by using a stabilization constraint; The apparatus further comprises: at least one third input adapted to receive, during a training phase, training movement sequence data (= training data) representative of a training movement sequence of the head and the at least one eye associated with a training environment, The at least one processor, during the training phase, - triggering, in a training environment, a series of head movements corresponding to said training movement sequence data; - assessing a current portion of a training environment from said training data and, if said current portion is not identified in said at least one memory, recording information regarding said current portion in said at least one memory; - determining information relating to the at least a portion of a stabilization constraint in function of the training data and the continuous movement of the head taking into account evaluating the current portion of the training environment and recording the information relating to the current portion, and recording the information relating to the at least a portion of a stabilization constraint in the at least one memory; and configured to learn at least a portion of the stabilization constraints by As a result, the virtual mammal provides a simulated behavior of the mammal to be simulated when performing the at least one locomotion behavior in the environment through continuous dynamically adjusted current movements of the head and the at least one eye, and through evaluating the environment.

[0011] In certain implementations, such devices may behave more or less similarly to real creatures, in this example mammals, thereby enabling accurate simulation of such mammals by virtual mammals, or equivalently mammal avatars. In this regard, such virtual mammals may enable, among other things, personalized simulation, monitoring, and prediction.

[0012] In some disclosed embodiments, this is obtained even more identically / precisely by such a device learning to behave in the same way as a real mammal through a deep learning process. The latter may have an advantage over other methods of producing such a virtual mammal in that it does not require the creation of a complete and accurate working model of the visual system or precise knowledge of the structures and functions involved in the real visual system. This deep learning process is advantageously performed by a neural network that is used to simulate at least a part of the visual system, and once the neural network is trained it can be used for the simulation.

[0013] In any case, the disclosed devices are not limited to visual aspects such as the neuronal aspects and ocular capabilities of an organism, but phototransduction capabilities can also be taken into account, either on their own or through more general conditions such as age, degeneration, or other physiological conditions.

[0014] In addition to other uses, the virtual mammals simulated by the disclosed apparatus may also be used as dummies for training ophthalmologists, surgeons, or other personnel involved in ophthalmology or vision equipment.

[0015] In addition, the control of dynamic adjustments of eye movements is not limited to the VOR mechanism: it can in particular involve intentional eye fixation and / or stabilization in reflexive visual movements (when following a moving object) instead or in combination.

[0016] During the training phase, learning at least a portion of the stabilization constraints may depend on implicit relationships between received training data, particularly representative of head movements on the one hand and eye movements on the other hand, these relationships depending on the stabilization constraints. The processor may take into account, in particular, stabilization mechanisms underlying images corresponding to at least one visual stream as seen by a mammal in an environment.

[0017] For example, during a training phase, head movements and at least a portion of such images are mimicked for a virtual mammal in an environment from received training data, the latter of which may include, for example, data directly related to such visual streams or data related to eye movements to which such visual streams are guided, in this example no eye movements need to be triggered when learning the stabilization constraints.

[0018] In another example, during the training phase, at least a portion of the head and eye movements are mimicked for a virtual mammal in an environment from the received training data, and the visual stream is then derived from that simulation.

[0019] Furthermore, during the training phase of the virtual mammal, environmental awareness can be taken into account in learning at least some of the stabilization constraints, making it possible to reflect an effective coupling between environmental awareness and the mammal-specific eye movement stabilization mechanism.

[0020] Indeed, the disclosed apparatus can potentially achieve substantial realism in incorporating combined environmental recognition and eye movement stabilization mechanisms, including a learning phase, an improvement that may prove particularly important for the simulation of visual systems.

[0021] The data representative of the eye posture and / or the training movement sequence data representative of the eye training movement sequence may in particular include data directed to the eye posture and / or one or more visual streams corresponding to the eye posture, which may in fact represent the eye posture or the eye training movement sequence.

[0022] In an advantageous implementation, the virtual mammal includes a movable body relative to the environment as well as a movable head.

[0023] The following features are also contemplated, which may optionally be used together with the inventive device, either alone or according to any technical combination of the following features: - the processor comprises software and computing hardware; - a training movement sequence of the head and the at least one eye is acquired on a simulated mammal, the training movement sequences of the head and of the at least one eye are computationally obtained and synthetic; The learning process during the training phase is carried out in a data-driven regime of the device, the device can be used in a data-driven regime capable of carrying out a training phase, The device can be used in a model-driven regime, if the current part is not identified in the at least one memory, instead of recording information about the current part of the environment in the at least one memory, the at least one processor is configured to replace previously recorded information in the at least one memory with information about the current part; - the recorded information about the current part of the environment in the at least one memory is erasable; The device is also for simulating pathological behavior, - The virtual mammals of the device are completely virtual, - the virtual mammal of the device has a substantial part; - the physical part includes a head, - the substantial part comprises at least one eye; - the entity part contains the body, - The solid parts include the limbs, -The head of the device is virtual, -The head of the device is a material element, At least one eye of the device is virtual; at least one eye of the device is a material element; - at least one eye of the device is capable of seeing; - at least one eye of the device is equipped with a camera; the instructions for the at least one movement action include an instruction to move the head; The instructions for the at least one movement action include an instruction to move the eyes; the instructions for at least one movement behavior include instructions to move both the head and the eyes; the instructions for the at least one movement behavior include an instruction to search for an object; The instructions for at least one movement behavior include an instruction to proceed to a predetermined location within the environment; "Pose" of the at least one eye means the position as well as the orientation of the at least one eye relative to a coordinate system; the "pose" of said at least one eye can be expressed as position and orientation values; the "pose" of said at least one eye can be represented as a vector, - the movement sequences of the head and the at least one eye are in successive postures over time; - data representing pose (position + orientation) relates to the eye pose itself or to the visual stream corresponding to the eye pose relative to the environment, -Stabilization constraints are related to the vestibulo-ocular reflex, - Stabilization constraints relate to intentional eye fixation, - Stabilizing constraints are related to the optokinetic reflex, the configuration for learning of the at least one processor is configured for performing a deep learning process with data representative of training movement sequences of a head and the at least one eye associated with a training environment in order to learn at least a portion of the stabilization constraints; -The mammal is a human being, - A mammal is an animal, - the simulated mammals are defined real individuals, The simulated mammal is a theoretical mammal; -Theoretical mammals are mammals that represent a group, -Theoretical mammals are statistically defined, - The theoretical mammal is a research mammal, During the training phase, the training environment in which the virtual mammal is located is preferably the same as the environment in which the training movement sequences of the head and of said at least one eye of the simulated mammal are obtained; during the training phase, the training environment in which the virtual mammal is located is not the same as the environment in which the training movement sequences of the head and of the at least one eye of the simulated mammal are obtained; Once the virtual mammal is trained in the training phase, the acquired environmental information / knowledge of the training environment is retained for further use of the virtual mammal; Once the virtual mammal has been trained in the training phase, the acquired environmental information / knowledge of the training environment is erased, so that the virtual mammal so configured starts to discover the environment from scratch; - The environment of the virtual mammal is realistic and - The environment of the virtual mammal is virtual, - to evaluate a current portion of said environment from said training data and, if said current portion is not identified in said at least one memory, to record information about said current portion in said at least one memory, which may also be considered as training of a virtual mammal device, said training being performed both in a data-driven regime and in a model-driven regime; at least one action of evaluating the current portion of an environment, recording information, and triggering the continuous movement is associated with at least one task parameter characteristic of the mammal to be simulated, the at least one processor being further configured for learning, in a training phase, the at least one task parameter in function of the training data and the continuous movement of a head, taking into account the evaluating the current portion of a training environment and the recording information regarding the current portion; The operating parameters are selected from any type of parameters that can be measured or derived from external characteristics, The task parameters relate, for example, to the movement of the eyelids in relation to the light level and the range of vision, The working parameters relate, for example, to the level of ambient light in the environment, the at least one processor further comprising: - obtaining at least one visual stream corresponding to said continuous data representative of said at least one eye pose, evaluating said current part of the environment from said at least one visual stream, triggering said continuous movement and controlling said dynamic adjustment in particular in function of said at least one visual stream, - configured during said training phase to obtain at least one training visual stream corresponding to said training movement sequence data representative of a training movement sequence, and to evaluate said current portion of a training environment and to determine said information in particular in function of said at least one training visual stream, the data representing the training movement sequence includes position and gaze direction data over time of a part of the virtual mammal (e.g., the torso and / or legs of the virtual mammal, etc.); - the at least one processor is further configured for conditioning the at least one acquired visual stream and the at least one training visual stream in relation to at least one conditioning parameter specific to the mammal to be simulated, the at least one processor is further configured for learning, during a training phase, the at least one conditioning parameter in function of the at least one training visual stream and the continuous movement of a head, taking into account the evaluation of the current portion of a training environment and the recording of the information relating to the current portion; the at least one processor further comprising: - deriving successive images from said at least one visual stream; - configured to control the dynamic adjustment by stabilizing at least a portion of the derived successive images; the at least one processor is further configured to control the dynamic adjustment by suppressing sequential movements of the at least one eye relative to the at least a portion of the sequential images; the apparatus comprises at least one fifth input adapted to receive information relating to a modified value of a stabilization constraint, the at least one processor being configured to replace information relating to a previous value of the stabilization constraint in a memory with information relating to the modified value; - the previous and revised values ​​of the stabilization constraints correspond to at least one of a pair of different mammal ages and a pair of different health states; the at least one processor comprises at least one first neural network configured to control the dynamic adjustment by a feedback loop; - said at least one first neural network configured for controlling said dynamic adjustment by a feedback loop is constructed based on a cerebellar vestibulo-ocular reflex control structure of a mammal to be simulated; - said at least one first neural network is a simulated neural network; at least one of the actions of evaluating the current portion of the environment, recording the information, triggering the continuous movement, and controlling the dynamic adjustment is associated with a task parameter, the device comprising at least one input adapted to receive reference motion data corresponding to the continuous movement of a head and the at least one eye, the reference motion data being associated with reference visual data corresponding to the received at least one visual stream, the at least one processor being configured to learn at least one of the task parameters from the reference motion data and at least a portion of the reference visual data; if a virtual mammal device learns at least one of said task parameters from at least a portion of said baseline movement data and said baseline visual data, it is in a data-driven regime; when the virtual mammal device has learned at least one of said task parameters from at least a portion of said baseline motion data and said baseline visual data, it is in a data-driven regime and is placed in a model-driven regime; at least one visual stream of the reference visual data is synthetic; at least one visual stream of reference visual data is acquired on the simulated mammal; at least one visual stream of reference visual data is derived from a reference gaze direction obtained on the simulated mammal; at least one visual stream of reference visual data is obtained from a camera positioned near an eye of the simulated mammal; the at least one processor comprises at least one second neural network configured to evaluate the current portion of an environment and record information about the current portion; at least one second neural network configured to evaluate the current portion of the environment and record information about the current portion is constructed based on a hippocampal structure of a simulated mammal, the hippocampal structure including a position coding; at least one second neural network is a simulated neural network; the at least one processor is configured to perform at least one of the following operations: evaluating, by simulated spiking neurons, the current portion of an environment; recording the information about the current portion; and controlling dynamic adjustment of continuous movements of the at least one eye; the apparatus comprises at least one fourth input adapted to receive at least one simulated visual characteristic, the at least one processor being further configured for dynamically conditioning the at least one visual stream in function of the at least one simulated visual characteristic; the at least one processor is further configured for dynamically conditioning the at least one visual stream in function of the at least one simulated visual property by virtue of at least one dedicated neural network simulating a simulated mammalian retina and early visual cortex; the dedicated neural network simulating the retina and early visual cortex is a simulated neural network; the at least one processor is further configured for dynamically conditioning the at least one visual stream in function of the at least one simulated visual characteristic by virtue of at least one customizable spatio-temporal filter for filtering the at least one visual stream; the at least one processor is further configured for dynamically conditioning the at least one visual stream in function of the at least one simulated visual characteristic by virtue of a set of cascaded customizable spatio-temporal filters; - Customizable spatio-temporal filters simulate visual instruments, - if the eye of the virtual mammal is a material element having a camera, a visual device, which is also material, is placed on the physical eye of the virtual mammal; - Customizable spatiotemporal filters simulate deformations of the eye, especially the eye's optical interface, - Customizable spatiotemporal filters to simulate eye diseases, - the at least one simulated visual characteristic is at least partially related to at least one physiological behavior associated with the at least one eye; - the physiological behavior is related to the corresponding functions of the eye itself or the visual cortex, said at least one simulated visual characteristic at least partially related to a visual instrument of said at least one eye;

[0024] A further object of the invention is to provide a process for simulating, in a device according to the invention, the physiological behaviour of a mammal in an environment by means of a virtual mammal, said mammal comprising: a head that is movable relative to the environment; at least one eye that is rotatably movable relative to the head, The process comprises: - receiving continuous data representative of a pose of said at least one eye relative to an environment; receiving instructions for at least one movement behavior by a virtual mammal in an environment; - evaluating, by at least one processor, a current portion of the environment from said continuous data using information about the environment stored in at least one memory, and if said current portion is not identified in said at least one memory, recording information about said current portion in said at least one memory; - triggering, by said at least one processor, a continuous movement of a head and said at least one eye in function of said at least one movement behavior, said continuous data and stored information on the environment; - controlling, by the at least one processor, a dynamic adjustment of the continuous movement of the at least one eye to the continuous movement of the head in function of the continuous data by using a stabilization constraint between the movement of the at least one eye and the movement of the head, The process further comprises: - receiving training data representative of training movement sequences of the head and the at least one eye associated with a training environment during a training phase; learning, by the at least one processor, during the training phase; - triggering, in a training environment, a series of head movements corresponding to said training movement sequence; - assessing a current portion of a training environment from said training data and, if said current portion is not identified in said at least one memory, recording information regarding said current portion in said at least one memory; - determining information regarding at least a portion of the stabilization constraints in function of the training data and the continuous movement of the head taking into account evaluating the current portion of the training environment and recording the information regarding the current portion, and recording the information regarding the at least a portion of the stabilization constraints in the at least one memory; and configured to learn at least a portion of the stabilization constraints by As a result, the virtual mammal provides a simulated behavior of the mammal to be simulated when performing the at least one locomotion behavior in the environment through continuous dynamically adjusted current movements of the head and the at least one eye, and through evaluating the environment.

[0025] The process can be carried out according to all the methods described, particularly for the above mentioned devices, possibly combined together.

[0026] The invention also relates to a computer program comprising instructions which, when executed by a processor, cause said processor to carry out the processes of the invention.

[0027] The computer program is advantageously configured for carrying out any of the disclosed process modes separately or in any combination.

[0028] The invention also relates to a non-transitory computer readable medium comprising computer program instructions which, when executed by at least one processor, cause said at least one processor in an apparatus to perform the processes of the invention.

[0029] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0030] [Figure 1] The major structural, functional, and environmental features and relationships of a virtual mammal simulated by an apparatus according to the disclosure are symbolically depicted. [Diagram 2] 1 shows a schematic diagram of an apparatus for simulating a virtual mammal as operating in a model-driven regime. [Diagram 3] 2 illustrates a schematic diagram of a preferred operational implementation of data and functions of an apparatus having functional modules. [Figure 4] 1 shows an example of different elements that may be considered to model / simulate the mammalian retina and early visual cortex. [Figure 5A] 1 shows an example of a mathematical model of a VOR controller that can be deduced. [Figure 5B] FIG. 1 shows a simplified structure of the mammalian VOR neural system being modeled / simulated. [Figure 6] We present a simplified mammalian hippocampal neuronal structure that is the basis for modeling / simulating the processing of visual and self-motion information during the construction of internal spatial representations. [Figure 7] 1 shows a schematic diagram of the material elements of an apparatus for simulating a virtual mammal as operating in a data-driven regime. [Figure 8] FIG. 2 is a flow diagram of the disclosed process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The disclosed device and process aims to simulate the physiological behavior of a mammal, which may be in a healthy or unhealthy state (it will be seen that additional functions and parts, organs may easily be added to the virtual mammal) using a virtual mammal, the mammal being simulated in the device. The simulated physiological behavior is thus derived from a healthy mammal or a mammal with some pathology. The mammal referred to is preferably a human being, especially when the device is applied to the selection of visual equipment, especially lenses or goggles.

[0032] The visual system in the virtual mammal is simulated using a set of functions, e.g., functions related to the action of the eye on the light entering the eye (e.g., related to the curvature and length of the eye), transformation of light / photons in the retina, different operations performed on the transformed light in parallel with different neuronal structure functions of the visual system of the real mammal that is the reference of the virtual mammal. These neuronal structure functions are, e.g., early visual processing structures, vestibulo-ocular reflex (VOR) neuronal structures. These functions are most preferably implemented / executed as neural networks in the virtual mammal to simulate real neuronal structures, with respect to those involved in the neuronal structures in the real mammal. However, if the function of the real neuronal structure is very simple and / or does not vary much between mammals, it can be more easily simulated by mathematical and / or logical functions that can be predetermined and fixed.

[0033] In the following description, the visual stream is considered as a vehicle / support of data representing pose (position + orientation). The visual stream corresponds to the eye pose relative to the environment and may be received within the device or determined by the device from known eye poses and from environmental data (the latter may be stored in the virtual mammal's memory / memories and is not the same as information about the environment that is related to information available to the mammal).

[0034] Moreover, the following description is mainly based on the control of dynamic adjustment of eye movements in the VOR mechanism. It can further include intentional eye fixation and / or stabilization with respect to the optokinetic reflex (when tracking a moving object), separately or in combination. Thus, each of these modes, VOR, intentional fixation, and optokinetic reflex, alone or in combination, can be implemented in the device (implemented with corresponding functional modules, as will be described later). At least the vestibulo-ocular reflex (VOR) is preferably implemented to simulate a virtual mammal.

[0035] 1 , a virtual mammal 1 simulated by an apparatus 11 according to the disclosure is located in an environment 2 and has a visual relationship with the environment 2, in the present example mainly in the form of a visual stream 10 of data corresponding to what is seen or supposed to be seen in the environment 2 by the virtual mammal's eyes 4, which corresponds to the information contained in the received light / photons. The eyes and part of the visual system process the visual information contained in the visual stream 10 to obtain environmental information 8.

[0036] The virtual mammal has a head 3 on which eyes 4 are located, and both head 3 and eyes 4 are movable as represented by rotating arrows 5, 6, with the eyes movable 6 relative to the head and the head movable 5 relative to the environment 2. These movements can be "voluntary" movements as opposed to reflex movements, with "voluntary" movements being under the control of movement behavior commands 9. "Voluntary" movements under the control of movement behavior commands 9 can be for "voluntary" movements of the head 3 or for "voluntary" movements of the eyes 4, depending on the type of movement behavior command. The eyes 4 are also subject to reflex movements in response to stabilization constraints 7.

[0037] The virtual mammal may be fully virtual or may comprise a material head that is movable with some material elements and material movable eyes, e.g. a camera, the whole being within a real and / or a material environment displayed on a screen. The virtual mammal may also be virtually placed within a fully virtual environment, especially if it is itself fully virtual. It may also comprise a body (fully virtual or with material elements) that may be movable within the environment. The virtual mammal may be implemented as a robot.

[0038] The material elements of the virtual mammal are in fact an extension of the simulated mammal. With regard to output / response, the simulated mammal also simulates the material elements, said simulation being used to control effectors that move or activate the material elements. The movement of the material elements is thus a result of the simulation, including in particular responses to voluntary or reflex movements through movement behavior commands. With regard to the input, in particular the visual stream, it may be synthetic (i.e. computer generated) or it may be obtained from a camera, in particular a camera at the eye position of the virtual mammal having a physical head and eyes. Of course, the simulated mammal may be extended by means of additional input and / or output eyes, arms, legs, hearing, etc., thus adding additional corresponding functional modules (see below).

[0039] Virtual mammals can therefore be highly detailed 3D models of real environments and can be placed into virtual environments or synthetic / virtual 3D environments that can be used to replicate recorded behaviors of mammals, particularly humans, and can be used to predict what a person for whom the virtual mammal is defined would do if placed in a similar real environment.

[0040] Here, Fig. 2 shows diagrammatically the main hardware elements implemented in a device for simulating the visual behaviour of a mammal in an environment. This Fig. 2 concerns a device in which a virtual mammal is easily constructed from a model of the mammal, which is then referred to as being implemented in a model-driven regime. The virtual mammal 1 is made up of a device 11 for simulating a mammal. The processor 12 is the heart of the device and can be implemented in several ways. It can be a computer, typically a microcomputer, an array or a network of computers, with at least one general purpose processor or a combination of a general purpose processor and special purpose processing means such as neuronal computing hardware or reprogrammable gate arrays. The processor 12 is under software control and communicates with a memory 13 and with its surroundings via input / output links. The memory stores in particular the environmental information 8 and the stabilization constraints 7. Again, the memory can be of any kind: general purpose or special purpose circuits such as RAM, EEPROM, etc., capable of storing information either physically or by the structure of the circuit, such as the neural circuit to be simulated.

[0041] The software on which the processor 12 runs is configured to simulate a mammal and is configured to simulate and control (via output links) head movements 5 and eye movements 6 in accordance with received commands (via input links) to movement behaviors 9 ("voluntary" movements) and the visual stream 10, including the processing / generation by the processor 12 of "reflex" eye movements to adapt the eye movements in accordance with stabilization constraints 7 and environmental information 8.

[0042] A thick arrow 611 is represented in Fig. 2 as a dashed line between the processor 12 and the visual stream 10, symbolizing the fact that the processor 12 can modify, in particular by conditioning, the received visual stream 10 based on defined conditions, in particular the visual characteristics to be simulated. The conditions can for example be related to the influence of lenses or goggles or some parts of the eye on the light supposed to be received by the eye and transmitted through them to the retina of the virtual mammal. For example, the conditions can be the interposition of a virtual lens on the virtual mammal's eye (or a real lens on the virtual mammal's real eye), the presence of degenerative tissue in the eye, the effect of age on the eye, the action of an ophthalmologist removing a part of the eye. For that purpose and to be able to select some of the conditions, a dedicated condition input link (not shown) can be added. The conditions can also be part of the software controlling the processor 12.

[0043] In FIG. 2, a regular arrow 56 is represented in dashed lines between the resulting head movement 5 and eye movement 6, which symbolizes the "reflection" part of the eye movement that is due to the head movement. Another regular arrow 610 is represented in dashed lines between the resulting eye movement 6 and the visual stream 10, which symbolizes the fact that when the eyes move relative to the environment, the information contained in the visual stream 10 changes, since the virtual mammal's eyes are no longer looking in the same direction. If the virtual mammal includes a moving eye that is a camera, the change of information in the visual stream 10 is automatic. However, if the virtual mammal's eyes are also simulated, this could be the processor 12 modifying the information in the visual stream, which could be done via the output 611 discussed above.

[0044] The model-driven regime of the virtual mammal is mainly driven by instructions for locomotion behavior to obtain goal-directed behavior. In the model-driven regime, the virtual mammal can be used to implement new behaviors. This model-driven regime can be used for prediction and monitoring.

[0045] Here, to build a virtual mammal in a model-driven regime, it may not be possible to specify a model from scratch. Moreover, it is often more advantageous to build a virtual mammal from knowledge gained about a real mammal (which may be, for example, a specific individual or an individual corresponding to a statistical individual or group) to obtain a virtual mammal that behaves similarly to a real mammal, and the behavior of the virtual mammal is built from the behavior of the real mammal through a learning process in a training phase of what is called a data-driven regime. The data-driven regime corresponds to the machine learning configuration of the disclosed device. Once the virtual mammal has been trained and learned behaviors (i.e., configured to behave as a real mammal through learning), it can be placed in a model-driven regime.

[0046] Thus, with the data-driven regime, as represented in Fig. 7, the software of the device for simulating a mammal is also configured to learn (represented by arrow 22) the stabilization constraints 7 from data 23 representing head and eye training movement sequences possibly associated with training environment information identified in the training phase of the data-driven regime. For that purpose, the processor 12 receives the head and eye training movement sequence data 23 and the training visual stream 10', while the virtual mammal is set in the identified training environment, which can be stored in the memory in the form of training environment information 8'. A database 21 can exchange data with the processor 12. The database can store parameters of the simulation, in particular recorded experimental data, which can be automatically loaded into the device. It will be seen that the database can furthermore store many other kinds of data, including predefined models, so as to directly configure the device in the model-driven regime, without the need to execute the data-driven regime and its training phase. They can also store the results of the simulation or the resulting models obtained from the training phase, for later reuse, without the need to redo the training phase later.

[0047] In this data-driven regime depicted in FIG. 7, the training movement sequence data 23 corresponds to applied eye and head movements, which are not and do not correspond to instructions for movement behavior as in the model-driven regime of FIG.

[0048] The training movement sequence data 23 representing eye training movement sequences associated with the training environment do not necessarily comprise eye movements, but instead may include relevant input visual streams as observed by a real mammal, an "average" mammal, or a fictitious mammal in the training environment, or other types of input visual streams for training.

[0049] For example, an input visual stream may be determined upstream from actual eye movements tracked by a suitable camera fixed on or external to the mammal and from the surrounding environment, and provided to the device as such, for example derived from gaze directions obtained from the tracked eye movements, through ray tracing in a virtual environment that corresponds to the surrounding environment, or from moving one or two cameras in a manner that reflects the tracked eye movements in the surrounding environment.

[0050] In another embodiment, the input visual stream is obtained by averaging several visual streams associated with different real people, e.g. belonging to the same group (e.g. same sex, similar age, similar eye disease, same height category, etc.) In yet another embodiment, the input visual stream is artificially created, e.g. based on a real-world visual stream that is subsequently transformed.

[0051] In any case, having eye movement information as training movement sequence data 23 is in fact a particularly interesting embodiment, with the associated training visual stream 10' being identified by the device from the received data of the training environment, for example by ray tracing associated with the eye pose corresponding to the eye movement information and with a virtual environment corresponding to the training environment, as is well known in the art.

[0052] In addition, the training movement sequence data 23 includes head and eye movement data corresponding to the training movement sequences associated with the training environment. Since the training movement sequences typically incorporate not only reflex eye movements but also environmental awareness, they typically take voluntary movements into account. In this regard, the environmental awareness process and the stabilization process may be closely intertwined and interdependent.

[0053] Mammalian eye movement stabilization properties are derived from a learning process. In this respect, head movement and visual stream, or head and eye movement, together with data related to the training environment may be sufficient. Information regarding stabilization constraints can in fact be derived from existing instabilities and stabilization effects in the acquired input visual stream, taking into account head movement and ongoing environmental awareness.

[0054] Note that in Fig. 7 (data-driven regime), there is no arrow between eye movements 6 and head movements 5, in contrast to Fig. 2 (model-driven regime) where there is an arrow. Indeed, in the training phase, eye and head movements are usually acquired and applied simply as received, e.g., in real-world situations, as observed above. They are simply acquired together as entries, and the absence of arrows in Fig. 7 means that the relationship between eye movements and head movements is not an issue when retrieving and applying these data in the training phase, so that their relationship is not usually considered in this regime.

[0055] In addition to the training phase of the learning process in the data-driven regime, the virtual mammal can be configured to reproduce experimentally recorded behavior to verify that the learned configuration or pre-established model works correctly. Moreover, in this data-driven regime, new data can be obtained that goes beyond the available experimental data. For example, if data on the user's head position in the environment, eye position relative to the head, the user's gaze direction and field of view can be experimentally measured, the virtual mammal simulation can additionally provide information on the content of the field of view, the visual cues the user is fixating on, and the neural activity of, for example, retinal ganglion cells.

[0056] When the stabilization constraints 7 are learned 22, the device can leave the training phase of the data-driven regime and return to its normal operating state in the model-driven regime, or can remain in the data-driven regime but not the training phase, for example to reproduce experimental behavior data. In the training phase, the learning process is typically performed through a deep learning process and / or any learning process that constitutes a neural network if such implementation is used. The use of training movement sequences of the head and eyes for learning corresponds to a data-driven regime in which the virtual mammal is asked to reproduce experimentally recorded behavior.

[0057] In other words, the training phase corresponds to a data-driven regime of the device, where the device learns to behave in the same way as the real mammal that it is to simulate. The resulting virtual mammal after the training phase is an image of the mammal that serves as a reference. The mammal that serves as a reference for training the virtual mammal may be a real mammal or a statistical mammal (e.g., the average of real mammals), or it may be a constructed, purpose-built mammal. The information addressed in the context of the term "learning" implicitly refers to database information that is learned and can be correlated, using appropriate analysis tools, with information resulting from the operation of the device (movement sequences, possibly associated locomotor behavior, possibly a visual stream induced in the training environment, etc.) with reference data on the behavior of the mammal, the database may in particular be available via a library that is either local (user library) or remote (server library).

[0058] During operation of the device in the learning phase or in normal operating conditions, it is possible to access the data processed in the device and also the behavior of the virtual mammal from the position of the real parts of the virtual mammal, for example the movable head and / or eyes. For example, it is possible to measure and / or record the position of the real head of the virtual mammal in the environment, the position of the real eyes relative to the head, the gaze direction and the field of view. Alternatively, it may be possible to obtain and / or calculate the same information from the processed data obtained from the device, especially if there are no real parts of the virtual mammal and one has to rely on the processed data. Moreover, if the function of the visual system is implemented / performed as a simulated neural network, it is also possible to obtain detailed information about the structure of the simulated / virtual neurons and how they behave, resulting from the learning process. It may also be possible to obtain information about how the visual stream is processed and "represented" in the visual system, and about the content of the visual field, the visual cues that the user is fixating on, and the neural activity of, for example, retinal ganglion cells.

[0059] The software controlling the processor 12 has many functions and it is preferable to implement a modular system for different functions, especially for ease of programming, maintenance, customization and upgrades, including adding new functions such as controlling the movement of an additional physical body of the virtual mammal. In addition, with a modular system it is easier to create a system that is more functionally similar to the real biological system of the mammal that is simulated in the virtual mammal.

[0060] Regarding the upgrade, as far as the visual system is concerned, and since the eyes are located on the head of the virtual mammal, the complete body movements, including walking, can again be followed by the head movements in the space of the environment, simplifying the process. In fact, it should be noted that, as far as the visual system is concerned, since the head supports the eyes and the vestibular system, considering the head movements in space may be sufficient for the simulation of the whole body.

[0061] In an upgraded implementation, the virtual mammal may have a complete body and may walk and / or perform other body movements that may be controlled by specific functional modules, as described below, resulting in head movements that are taken into account by a functional module directly related to the visual system.

[0062] The system can therefore be easily expanded by adding modules that implement additional functions. Each functional module simulates a biological organ or brain area or part thereof (e.g., the eye, the retina and visual cortex, the hippocampus; the cerebellum; the musculoskeletal system, and the control areas in the brain that control the balance and posture of the virtual mammal). Each function can be implemented by several functional modules at different levels of biological detail. To speed up the simulation, some functional modules can be implemented by fast algorithms, while others can be implemented by detailed biological simulations involving neural networks or cell models.

[0063] 3 is an exemplary modular system with communication means to which different components such as functional modules, processing means, auxiliary tools and data structures are interconnected. This modular system with its components is under the control of a control program that is part of the device's software. More precisely, in this embodiment: a server library component 14, a user library and analysis tools 15, a 3D aging virtual mammal and its environment component 16, and three functional modules 17, 18, 19 are connected to a message passing middleware 20. The three functional modules represented are: a virtual retina and early visual cortex module 17, a cerebellar VOR control module 18, and a hippocampus module 19.

[0064] The 3D Aging Virtual Mammal and its Environment component 16 refers to the word "aging" because it has been developed to take into account the age of the virtual mammal, and the capabilities and functions simulated by the device that evolve as the virtual mammal ages.

[0065] The Server Library component 14 manages the components (registration, subscription, interface, broadcast). The 3D component 16 is a specific client module that performs the visualization of the virtual mammal, its environment and selected behaviors using a 3D simulation environment. The User Library and Analysis Tools component 15 (including analysis and graphic user interface) performs the control and analysis of the simulation platform and its main functions include: - Configuration of virtual mammal parameters (either manually, or loaded from a file, or learned). Configuration of virtual mammal environments that can be loaded, for example, from a file. - Configuration of the virtual mammal behavior in terms of functional modules. The user can for example load experimental data that determines the user behavior. In a data-driven regime, where a learning process is performed, the behavior is specified by experimental data, for example head position and gaze direction as a function of time. In a model-driven regime, the behavior is specified by an experimental protocol that describes the behavioral task (for example, in a navigation experiment, the starting position and the goal must be given, as well as the number of learning trials). - Access to any specific data analysis tools developed to process and visualize the results of the simulation. These analysis tools may relate to specific components of the virtual mammal, the evaluation of its behavioral performance, etc.

[0066] When the functional modules and the virtual environment are known, it is possible to efficiently parameterize the virtual mammal of the device. The parameters of the model can be separated into global parameters (such as, for example, age, sex, specific medical conditions, etc.) that affect all or most of the functional modules and components, and local parameters (such as, for example, visual acuity or contrast sensitivity, or any other experimentally measured parameter of the mammal) that affect the behavior of a particular functional module or component.

[0067] The message passing middleware 20 is the backbone of the system that allows each component connected to the system to send messages to other components and to invoke functions implemented on other components. From a technical point of view, the implementation of the device can be done via a distributed system that can run on one computer, on several computers or on a computer cluster, depending on the number of modules and their complexity.

[0068] As mentioned above, other components may be added, which may be functional modules that contain computational models of specific brain regions (e.g., the primary visual cortex) or biological organs (e.g., the eye). Each functional module must be registered with the system, can subscribe to messages of other modules, and must expose its own functionality.

[0069] The implementation of the message passing middleware 20 may use the open source ROS framework (http: / / www.ros.org). In particular, a development of this framework using the C / C++ language and called cROS (https: / / github.com / rrcarrillo / cros) may be used. This development includes a ROS master and a server module that manages the components using a standardized software interface and implements message passing / broadcasting and remote function calls. In a biologically plausible implementation of the functional modules, these modules exchange messages represented by spike trains, i.e. short sequences of events that simulate the electrical action potentials used by neurons in the nervous system to exchange information. For this purpose, special types of messages have been compiled into the ROS library so that any module can asynchronously send a spike train to any other module, modeling information transmission in the brain. The system can also be extended by adding other custom message types.

[0070] The 3D Aging Virtual Mammal and its Environment component 16 may be developed using the "Unity 3D Engine" (https: / / unity3d.com / unity). The "Unity 3D Engine" provides many features that can be used to model realistic 3D environments, including lighting effects, physics, and animations. Full control of the engine can be performed using the programming language C#. This functionality is used to remotely control the 3D model from the User Library and Analysis Tools 15 component. The subcomponents of the User Library that implement the control of the 3D Aging Virtual Mammal and its Environment 16 component are referred to herein as the Control Program.

[0071] Five general C# scripts are used to control the 3D aging virtual mammal and 16 components of its environment. -RosInterface: This script implements the ROS interface so that the 3D aging virtual mammal and its environment components can exchange messages with other modules, including the control program from which the 3D aging virtual mammal and its environment components can be remotely controlled. -SimulationManager: This script is called by the control program and is responsible for managing the state of the simulation. It is used to create the virtual mammal and its environment, as well as to start, pause and stop the simulation of actions. -SubjectMovement. This script executes the behavior of the virtual mammal as commanded by the control program. This script also receives raw sensory data from the virtual mammal sensors (e.g., receives the flow of visual information in the visual stream experienced by the virtual mammal). This sensory information is returned to the control script or to any other modules that subscribe to receive it via the subscription service of the server library component. -SubjectEye. This script implements the eye of the virtual mammal, which can be simulated by a video camera attached to the virtual mammal's head. -PALens. This script performs distortion of the visual input with any visual device attached to the virtual mammal's eyes: single vision, lenses / goggles, filters, photochromics, etc. This is what the script is conditioning the visual stream.

[0072] Thus, the 3D aging virtual mammal and its environment component 16 is used to visualize the virtual mammal, its environment, and its behavior. In this embodiment, this component is also responsible for simulating the visual distortion effects due to visual equipment such as lenses / goggles.

[0073] For the visualization of the virtual mammal's environment, any detailed 3D model of the environment created by dedicated software (such as AutoCAD®) can be loaded as the environment model. In another implementation, a 3D model of the actual laboratory can be developed directly in the Unity 3D engine.

[0074] The visualization of the virtual mammal head and body, and more generally its physical appearance and its movements, if humanoid, can be implemented using the MakeHuman software (http: / / www.makehumancommunity.org). The virtual mammal may be a customizable 3D model of a generic human, if the simulated mammal is human. In the current implementation, the customization of the visual appearance includes age, sex, and outlook. However, any detailed 3D human model (e.g., created by dedicated software) can be loaded.

[0075] The User Libraries and Analysis Tools 15 component allows the user to: - Loading virtual mammal parameters from a parameter file or an experimental database. - Creating, starting and stopping simulations during normal operating conditions or during the training phase, using a control program written, for example, in Matlab or Python. - Obtaining simulation data generated by any module of the platform, specifically by subscribing to messages from the corresponding module, and analyzing this data to provide performance evaluation, monitoring, and prediction.

[0076] We now describe some of the functional modules that can be used, more particularly those related to the visual system and the adaptation of the eyes to head movements. The functional modules considered are those represented in Figure 3 and implemented in relation to different functions of a real mammalian brain, with regard to its visual system and the adaptation of the eyes to head movements.

[0077] The virtual retina and early visual cortex module 17 is a functional module that simulates the activity of retinal ganglion cells in the mammalian, in particular human, retina. Its functional module is based on the elements represented in FIG. 4. It can also be applied to simulate orientation-sensitive neurons in the V1-V2 area of ​​the brain. As input, it receives the visual stream from the virtual mammal's eye or from the camera of the virtual mammal's real eye. As output, it generates spike trains (sequences of simulated action potentials) emitted by the simulated retinal ganglion cells. The retinal processing is simulated in a detailed manner as a cascade of customizable spatiotemporal filters applied to the retinal input visual stream. The parameters of this model, such as the properties of the spatiotemporal filters, are pre-specified or learned and can therefore be adjusted to the retinal recording data (usually performed with an extracted retina on a dish with measuring electrodes so that the retinal activity is recorded). In particular, age-related changes in early visual processing can be taken into account in this model by parameterizing the virtual retina model.

[0078] The cerebellar VOR control module 18 is a reflex eye movement that stabilizes the image on the retina during head rotation by contralateral eye movements with the aim of keeping the image in the center of the visual field. The VOR is mediated by the brain region where adaptation is directly driven by sensorimotor errors, namely the cerebellum. The cerebellar VOR control module 18 may implement a detailed model of VOR control by neural networks in the cerebellum, as based on the neuronal structure depicted in FIG. 5B. In a more simplified implementation, the cerebellar VOR control module 18 may be implemented based on the controller model depicted in FIG. 5A as a mathematical function.

[0079] The hippocampal module 19 is based on a neuronal model of the hippocampus and is responsible for constructing a neural representation of the virtual mammal's environment. This neuronal model is obtained from the representation in FIG. 6. Vision-based spatial navigation in mammals is controlled by a neural network in a brain region called the hippocampal formation. Several types of neurons in this brain region, particularly place cells and grid cells, interact in a complex way to construct a neural representation of the surrounding environment based on the incoming sensory input. As represented in FIG. 6, self-motion input and allocentric visual input enter an array of grid cells, EC, and the processed information is routed through other arrays of neurons, namely the DG for feedback (sub-alpha), the CA3 with visual place cells and movement-based place cells, and the CA1 with associative place cells. The hippocampal module 19 simulates, by means of a neural network, the operation of the hippocampal formation during spatial evaluation of the environment by the hippocampal formation.

[0080] The hippocampus module 19 receives as input the following pieces of information: - Visual information as output by the virtual retina and early visual cortex module 17 receiving the visual stream. -Other sensory information such as proprioception, vestibular etc.

[0081] The hippocampal module 19 generates as output neuronal activity that encodes the location of the virtual mammal within the environment. These outputs can then be used to drive goal-directed behavior of the virtual mammal in a model-driven regime.

[0082] The virtual mammal of the device is capable of evaluating the current part of the environment from the visual stream it receives and is also capable of recording information about said current part in its memory, which recording can be considered as a kind of learning / training. The device can thus be considered to have a dual training scale: one level is directed to configuring the virtual mammal for dynamic eye movement coordination in a data-driven regime with its training phase, another level is directed to learning the environment in a data-driven as well as a model-driven phase, otherwise the data-driven phase involves a combined dual training scale.

[0083] Regarding the two possible regimes of the device, the data-driven regime can be executed after the model-driven regime has already been applied, in particular to reconstruct the virtual mammal (e.g., considering another individual, changing the health state, or aging the same individual, etc.), and in any case it precedes the model-driven regime based on it and is preliminary to this extent.

[0084] In a data-driven regime, the movement of a virtual mammal within the environment is performed based on commands issued from a control program.

[0085] In one implementation, the control commands include a set of recorded spatial coordinates of body parts as a function of time that the virtual mammal must reproduce (e.g., provided by motion capture equipment). To facilitate visualization, the control movements are interpolated to provide the basis for the final virtual mammal animation, which is created by mixing a standard set of five base animations.

[0086] With respect to the movements of different parts of the virtual mammal, the control program also provides gaze direction data that is used to control the virtual mammal's eyes. In one implementation, spatial coordinates of the gaze target in the 3D environment are provided as a function of time. An inverse kinematics approach is used to match the movement of the virtual mammal's head with the movement of the virtual mammal's eyes. The inverse kinematics equations identify a coupled eye-head position that matches the data and the constraints of the 3D model of the virtual mammal. Additional parameters such as eye wavefront, pupil diameter, and eyelid movement (obtained by an eye tracker) can be added to improve the accuracy of the simulation. The gaze direction data and additional parameters can also be used in the training phase for learning the stabilization constraints.

[0087] In the model-driven regime, the control program provides an experimental protocol of the behavioral task to be performed. In this case, the movement of the virtual mammal is controlled by an additional functional module that implements the brain regions that perform the movement control. In fact, the movement control can be implemented in many ways. For example, a specific module for the movement control, i.e., a movement control module, can be added to the modular system. In another implementation corresponding to the modular system of FIG. 3, with VOR adaptation, the cerebellar VOR control module 18 implements the eye movement adaptation by directly controlling the eye position for the 3D aging virtual mammal component 16.

[0088] In another implementation, an additional functional module can be added to the modular system, namely a model of an eye muscle module controlled by spiking the output of the cerebellar VOR control module 18, which implements eye control in a highly detailed and physiologically plausible manner. In the same way, but on a larger scale, the hippocampal module 19 in the implementation of FIG. 3 can also directly control its own movement, if the virtual mammal is constructed to move, by simply setting the direction and speed of the virtual mammal's movement. Alternatively, a specific locomotion module can be added to the modular system that implements motor control of the virtual mammal in a physiologically plausible manner.

[0089] The same interpolation algorithm is used to implement natural behavior of virtual mammals.

[0090] An example of a simulation in the model-driven regime is shown below. 1. First, a simulated mammalian subject is recorded during some experimental task to generate recorded data. For example, adaptation to a new visual device is recorded by measuring eye movement dynamics over several trials in a defined environment. 2. The virtual mammal is then simulated in a data-driven regime to learn the model parameters of at least one functional module involved in the eye adaptation movement, in this example the cerebellar VOR control module with its stabilization constraints, which is based on a neural network. For that purpose, the virtual mammal currently in the training phase is simulated using the previously obtained recording data and with the same corresponding environment. When the functional model is trained, and especially when the stabilization constraints are set, it represents a model of how this particular subject adapts to the new visual equipment. 3. After the functional module has been trained in step 2, it can be used to predict how the subject will adapt to another visual device not previously tested, or possibly the same one as a means to check the quality of the model. In this model-driven simulation, the virtual mammal parameters, in particular the stabilization constraints in this example, are therefore fixed and the new visual device is installed on the virtual mammal. The simulation is then started and the performance of the virtual mammal is recorded. This performance constitutes a prediction of what the subject will do if this new visual device is fitted to the subject in this particular task.

[0091] More generally, the same sequence of data acquisition (to obtain recorded data from the mammal to be simulated and the associated environment), model training (the learning process during the training phase of the virtual mammal behavior), and model-based prediction can be used with any functional module. For example, the cerebellar VOR control module 18 can be trained on recorded data to predict how the subject will adapt to new experimental conditions. As another example, the hippocampal module 19 can be trained through a learning process on a real navigation task in one environment. In a model-driven regime, the virtual mammal can be used to predict what will happen if the same navigation task is performed in another environment. Any module capable of representing adaptation and learning can be trained and then used for prediction.

[0092] In particular, with regard to the visual system simulated in the virtual mammal, the mammal's eye is simulated as a cylindrical camera, which may be virtual (the virtual mammal has no solid part simulating the eye) or real (the virtual mammal has a solid part simulating the eye and is equipped with a camera).

[0093] Additionally or alternatively, any detailed model of a mammal, particularly a human eye, can be used, which can be implemented, for example, by a modified SubjectEye C# script to perform the desired calculations directly by the 3D aging virtual mammal module. If the model is highly complex and requires dedicated processing, a new functional module can be implemented, which may run on a separate computer or computer cluster, to process the visual stream and expose it to other functional modules using a message passing middleware.

[0094] Similarly, any visual instrument can be modeled for the virtual mammal's eye, either by implementing it directly in the 3D aging virtual mammal module with C# scripts or by using a dedicated function module. It is possible to simulate visual distortions and / or add some aberrations (on the wavefront) to the eye's lens and calculate the effect on the retinal image. It is then possible to evaluate the effect of astigmatism and higher order aberrations on the subjective best focus of the virtual mammal.

[0095] The implementation of a virtual mammal using the modular system can be carried out according to the following method. 1- Definition of a virtual mammal based on the simulated mammal. The definition includes parameters related to the mammal profile. It takes into account demographic information such as age and / or sex and / or other information. As an example of other information, in the context of ophthalmic use, parameters such as at least prescription, fitting parameters, but also sensory, kinematic, movement and cognitive state information can be taken into account. The definition content depends on the purpose and the context of the simulation. The definition can be entered in many ways: manually, using a file with the relevant parameters or learned during a training phase. In this definition, the goals and problems to be solved by the device for simulating a virtual mammal are also set. The goals and problems can be related to any field, industrial, medical, gaming and include at least: Learning during the training phase to set parameters such as stabilization constraints throughout the learning process. - Playing / rendering movements with virtual mammals as a result of learning. -Evaluation or monitoring of movement or other elements (e.g. the representation of visual information in parts of a modeled (neural network) visual structure) or, more generally, data (in particular stabilization constraints, visual quality, etc.) and / or structures (in particular the structure and properties of neural networks) involved in processing in the device. -Predicting movements or other elements. 2- Functional module selection: The selection of regime (model-driven regime or data-driven regime) and modules is done according to the user's objectives. 3- The output / response of the 3D virtual mammal simulation can be sent to one specific user or a cluster of users. The output / response can be sent from different functional modules and / or further processed to obtain an assessment of behavior such as navigation, autonomy, and sensory, motor, and / or brain levels. 4- Identification / selection / optimization of solutions to problems faced by the virtual mammal. This is a neurocomputation-based method / process for identifying or selecting objectively or subjectively any problem with the simulation, reproduction, visual equipment lens, or for providing personalized recommendations to improve the health of the mammal. The effects of visual equipment or re-education / functional restoration can be simulated and tested in a virtual environment that models the performance of the mammal. Thanks to software modeling, the behavior of an aging mammal can be simulated or reproduced to predict the effects of potential visual processing (e.g. eye disease), human brain, behavioral deficiencies, or poor vision (e.g. uncorrected or improper refraction). In the standalone version, the virtual mammal includes an automatic evaluation mode to compare the output / response of the virtual mammal. This standalone version provides a benchmark for establishing recommendations regarding the best solution.

[0096] The applications of the disclosed device are numerous.

[0097] In a first application, it can be used to select new lenses / goggles by evaluating their effect on the locomotion of a simulated mammal. In this first application, the device simulates the distortions caused by the lenses / goggles on the visual stream reaching the retina. The movement of the real mammal is recorded using a motion and eye-tracking system during exploration of a real environment. The data recorded by the system consists of the spatial coordinates of each sensor and eye-tracking data. These data are loaded into the device during a training phase via a control program in order to control the movement and gaze direction of the virtual mammal for learning in a data-driven regime. After the training phase, a learned virtual mammal is created wearing the lenses / goggles throughout the simulation, and its distortions due to visual input are modeled using PALens scripts in the 3D aging virtual mammal component 16. The user library and analysis tools component 15 allows visual inputs with and without the lenses / goggles to be compared with each other during exploration of a simulated real-world environment. In particular, visual distortions can result in changes in balance and locomotion patterns. Therefore, by analysing the changes in visual input caused by visual devices and linking these changes to postural control balance, sway and walking speed, it is possible to adjust the characteristics of the visual device to minimise its impact on performance.

[0098] In this first application of the disclosed device, it can be used to select from among a database of existing eye drops an eye drop that fits the wearer's daily behavior, such as mobility, navigation activities, etc. The definition of the human virtual mammal includes the wearer profile (prescription, age, fitting parameters, etc.). For example, the parameters in the definition of the virtual mammal may be: 65 year old individual who needs a prescription change, current lenses have a short progressive length; The environment is a commercial environment; The input parameters are an eye drop database that includes the available visual instruments and a data record of the body movements recorded during a simplified navigation task in a commercial environment. For functional module and component selection, when the virtual mammal wears different visual instruments, their distortions on the visual stream input are modeled using PALens scripts in the 3D Aging Virtual Mammal and its Environment component. According to the User Libraries and Analysis Tools component 15, the minimum configuration of the involved modules and components is as follows: - For the data-driven regime, a 3D Aging Virtual Mammal and its Environment component 16 with its PALens script for the calculation of visual environment distortions, and a Cerebellar VOR Control Module 18. - for a model-driven regime, a 3D aging virtual mammal and its environment and functional modules, including a cerebellar VOR control module 18 for recalculating eye movements based on head movements recorded with the current lens. In effect, the wearer, having adapted to his / her lens distortion, presents an optimally simulated vestibulo-ocular reflex (VOR) to stabilize the image on the retina.

[0099] Regarding the output / response of the virtual mammal, in a data-driven regime, the output / response parameters are distortion quantities calculated from recorded movement patterns, eye movements, and a simulated commercial environment. The literature, WO2017157760A1, can be referred to as a method for performing a selection of visual instruments that minimizes lens distortion on the retinal visual stream.

[0100] In terms of the output / response of the virtual mammal, in a model-driven regime, this gives rise to a complementary evaluation by inverse engineering methods. A wearer with a prescription of +2.50 to +3.00 dp additional changes faces new visual motion due to the distortion and has to adapt his / her vestibulo-ocular reflex (VOR) response. The sensorimotor error caused by the distortion of the new visual equipment is calculated by using the PALens scripts of the functional modules Cerebellar VOR Control Module18 and Virtual Retina and Early Visual Cortex Module17, as well as the 3D Aging Virtual Mammal and its Environment component16.

[0101] For the determination / selection / optimization of a visual device or intervention solution, the user of the device selects a new lens among the available existing designs with an addition of +3.00 dp that produces the minimum sensorimotor error of the Vestibular Ocular Reflex (VOR) response compared to the wearer's current visual device with +2.50. Those skilled in the art know that the distortion of the visual device on the temporal side of the lens is stronger in short designs or to present a "soft" or "hard" lens design. However, the current approach defined the hardness of the lens from an optical calculation of the local rate of change of the lens' undesirable astigmatism and the associated optical deviation (local prismatic effect). This change is more or less steep and complex. However, the lens distortion is caused by several factors such as: the fitting parameters, the optical design, as well as the wearer's ocular head pattern during movement. In the new approach, a model-driven regime gives the user of the device the opportunity to have a wearer approach in evaluating the Vestibular Ocular Reflex (VOR) mechanism generated by the spontaneous movements and eye movement patterns of each wearer. The personal selection is made from a database of available lenses by comparing the distortion of the current visual device with all visual devices that have been tested on the wearer.

[0102] In a second application of the disclosed device, new visual instruments can be designed using the vestibulo-ocular reflex (VOR) time adaptation aging effect as a performance criterion during the conception stage, which is the manual or automatic modification of lens parameters such as power and subdivision of unwanted astigmatism, definition of the front and back surfaces, etc. In this application, the age effect on the vestibulo-ocular reflex (VOR) adaptation experiment is simulated.

[0103] First, a virtual mammal is constructed using standard parameters of a young human mammal. For the construction, the device of the virtual mammal is trained in a training phase using a learning process, and data obtained on the young mammal performs at least a sinusoidal rotation of the head in the horizontal plane, using a specified environment. During an experiment with the virtual mammal of the device, a sinusoidal rotation of the head of the virtual mammal in the horizontal plane is induced using the same environment. During this experiment with the virtual mammal, the cerebellar VOR control module 18 automatically adjusts the control of the "eye muscles" of the virtual mammal to counteract the sinusoidal movement of its head by an opposing movement of the virtual mammal's eyes. Vestibulo-ocular reflex (VOR) adaptation based on image slip on the retina is evaluated by an analysis of the dynamics of the eye movements of the virtual mammal. The amplitude of the eye movements due to insufficient stabilization must decrease over time. The time scale of the adaptation is the measurement variable.

[0104] The same operation is then performed by an aged virtual mammal in which age-induced changes in the VOR adaptability are set. These age-induced changes result in a slowdown in the VOR adaptability dynamics, which can be analyzed by the User Library and Analysis Tools 15 component. For VOR adaptability, if necessary, reference can be made to "Adaptation to Telescopic Spectacles: Vestibulo-ocular Reflex Plasticity" Demer et al., 1989 IOVS and "Effect of adaptation to telescopic spectacles on the initial human horizontal vestibulo-ocular reflex." https: / / doi.org / 10.1152 / jn.2000.83.1.38.

[0105] In practice, the choice of regime model is based on available data and wearer parameters. The wearer profile (prescription, age, fitting parameters, etc.) may be a young ametropic patient and a 40-50 year old presbyope, or a 40-50 year old presbyope, or a 60-70 year old presbyope, or a 70-80 year old presbyope. The environment is defined. It can be performed using behavioral data (data-driven regimes) or tasks (model-driven regimes).

[0106] In the model-driven regime, a new visual device is designed based on standard parameters of juvenile mammals used to construct a virtual mammal through learning in a training phase, while estimating the impact of this new design on a criterion defined by vestibulo-ocular reflex (VOR) time adaptation with age.

[0107] Again, the input data are standard parameters of a young mammal and are used to construct the virtual mammal, for which purpose a sinusoidal rotation of the virtual mammal's head in the horizontal plane is induced.

[0108] A selection of functional modules and components is made. The virtual mammal is fitted with lenses / goggles and its distortion on the visual stream is modelled using the PALens script in the 3D Aging Virtual Mammal and its Environment 16 component. According to the User Libraries and Analysis Tools 15 component, the minimal defined configuration of involved functional modules and components is as follows: -For the data-driven regime: PALens script, a 3D aging virtual mammal and its environment16 components with a virtual retina and early visual cortex module17. -For the data-driven regime: 3D aging virtual mammal and its environment 16 components, a virtual retina and early visual cortex module 17, a cerebellar VOR control module 18.

[0109] With respect to the virtual mammal output / response, the time scales of residual retinal slip and adaptation output parameters are calculated for all age groups as a result of age-induced changes. The rates of vestibulo-ocular reflex (VOR) adaptation and residual retinal slip are identified for each age group.

[0110] With regard to determining, selecting, optimizing a visual device or selecting an intervention solution, multifactorial optimization consists of identifying design parameters of the visual device such as astigmatism, magnification, length progression, etc. that reduce the time-cost adaptation of the vestibulo-ocular reflex (VOR) and residual retinal slip, taking into account the amplitude and velocity of eye movements per age group.

[0111] In a third application of the disclosed device, the influence of visual equipment on a mammal's ability to estimate its own motion trajectory from the visual stream 10 can be evaluated. The movements of a real mammal are recorded using a motion and eye-tracking system during a navigation task (either in a real environment or in a virtual environment with a head-mounted display), where the mammal is required to follow a predefined trajectory from a starting position (e.g., two sides of a triangular trajectory) and then return to an unmarked starting position. This task is repeated over several trials. Returning to the starting position requires estimating one's own position relative to the start, which is known to depend on neuronal processing in the hippocampus. The accuracy of the self-location estimation is determined from the error made by the subject when returning to the starting position.

[0112] In the virtual environment, several different kinds of visual inputs can be simulated to evaluate the lens distortion depending on the statistics of the surrounding visual cues. The same task in the absence of visual input allows the estimation of the self-location based only on the proprioceptive input (as a vision-independent control condition). The recorded data consists of the spatial coordinates of each sensor and the eye-tracking data. A virtual mammal is created, including a virtual retina 17, a VOR control 18, and a hippocampal module 19. The data is loaded into the device during the training phase via a control program in order to control the movement and gaze direction of the virtual mammal for learning in a data-driven regime. As a result of the learning, the neural network of the hippocampal module 19 obtains a representation of the position with a precision adjusted to that of the real mammal.

[0113] Visual distortions are introduced into the model-driven regime via simulated visual equipment using PALens scripts in the 3D aging virtual mammal module. The impact of the visual distortions introduced by the lenses on the performance of the tasks is analyzed by the User Libraries and Analysis Tools component 15. Apart from testing the impact of distortions introduced by the lenses, changes in task performance related to the progression of ageing or visual diseases can be tested. In particular, considering several longitudinal measurements of a healthy or pathological mammal's visual system (e.g., its retinal condition, useful visual field, light diffusion in the eye, etc.), a model of age-related or disease-related changes in visual processing over time can be built. This model can be loaded into a virtual mammal and then used to predict the state of the mammal's visual system in the near future. Such an aging virtual mammal model allows simulating the future behavior of the mammal (and its adaptation to new environmental conditions or to new visual equipment) and providing potential rehabilitation solutions or proposing new visual equipment to counteract age- or disease-related problems.

[0114] 8, a flow diagram of the disclosed virtual mammal operation process, illustrates two possible ways of operating a virtual mammal: a model-driven regime and a data-driven regime, this last regime operating in a learning phase to learn at least some information of the stabilization constraints.

[0115] In a data-driven regime, the steps are: receiving training data representative of a head and at least one eye training movement sequence associated with a training environment; learning at least a portion of said stabilization constraints by: - triggering, in a training environment, a series of head movements corresponding to said training movement sequence; -Evaluating a current portion of a training environment from said training data; and - testing whether the current part is not identified in at least one memory; - if not, recording information about the current part in at least one memory; - determining information about at least a part of a stabilization constraint in function of the training data and the continuous movement of the head, taking into account an evaluation of a current portion of a training environment and a recording of information about said current portion; - recording information relating to at least a portion of said stabilization constraints in at least one memory; and learning by

[0116] At the end of these steps, at least a portion of the stabilization constraints are recorded and can be used in the model-driven regime, which is symbolized by the discontinuous arrow from the last step of the data-driven regime to the model-driven regime.

[0117] In a model-driven regime where stabilization constraints exist in the system (either learned from a previous data-driven regime or obtained by other means), the steps are: - receiving continuous data representative of a pose of at least one eye relative to an environment; receiving instructions for at least one movement behavior by a virtual mammal in an environment; - estimating a current portion of the environment from the continuous data using information about the environment stored in at least one memory; - testing whether the current part is not identified in at least one memory; otherwise, recording information about the current portion in at least one memory; triggering a continuous movement of the head and at least one eye in function of at least one movement behavior, said continuous data and stored information on the environment; - controlling the dynamic adjustment of the continuous movement of at least one eye to the continuous movement of the head in function of the continuous data by using a stabilization constraint between said at least one eye movement and the head movement.

Claims

1. A device (11) for simulating the physiological behavior of a mammal in an environment (2) by means of a virtual mammal (1), said virtual mammal (1) comprising: a head (3) that moves (5) relative to said environment; at least one eye (4) that moves in a rotational manner (6) relative to the head (3), The device (11) comprises: at least one first input adapted to receive continuous data representative of a pose of said at least one eye (4) relative to said environment (2); at least one second input adapted to receive commands for at least one movement behavior (9) by said virtual mammal (1) in said environment (2); at least one memory (13) adapted to store information about said environment (8) and about stabilization constraints (7) between the movements (6) of said at least one eye (4) and the movements (5) of said head (3); at least one processor (12), evaluating a current portion of said environment (2) from said continuous data and, if said current portion is not identified in said at least one memory (13), recording information (8) relating to said current portion in said at least one memory (13); triggering a continuous movement (5, 6) of the head (3) and the at least one eye (4) in function of the at least one movement behavior (9), the continuous data and the stored information about the environment (8), at least one processor (12) configured for controlling dynamic adjustment of the continuous movement (6) of the at least one eye (4) relative to the continuous movement (5) of the head (3) in function of the continuous data by using the stabilization constraints (7), The device (11) further comprises: at least one third input adapted to receive, during a training phase, training movement sequence data (23) representative of a training movement sequence of said head (3) and said at least one eye (4) associated with a training environment, The at least one processor (12) performs, during the training phase: - triggering, in said training environment, a sequence of movements (5) of said head (3) corresponding to said training movement sequence data (23); - assessing a current portion of the training environment from said training movement sequence data (23) and, if said current portion is not identified in said at least one memory (13), recording information about said current portion (8') in said at least one memory (13); - determining information relating to at least a portion of the stabilization constraints in function of the training movement sequence data and the continuous movement of the head taking into account the evaluating the current portion of the training environment and recording the information relating to the current portion, and recording the information relating to the at least a portion of the stabilization constraints in the at least one memory; configured to learn at least a portion of the stabilization constraints (7) by As a result, the virtual mammal (1) provides, through the continuously dynamically adjusted current movement of the head (3) and the at least one eye (4) and through evaluating the environment (2), a simulated behavior of the mammal to be simulated when performing the at least one locomotion behavior in the environment. Device (11).

2. 2. The device (11) of claim 1, characterized in that at least one of the actions of evaluating the current portion of the environment (2), recording the information (8, 8') and triggering the continuous movement (5, 6) is associated with at least one task parameter specific to the mammal to be simulated, and the at least one processor (12) is further configured for learning the at least one task parameter during the training phase in function of the training movement sequence data and the continuous movement (5) of the head (3), taking into account the evaluating the current portion of the training environment and recording the information regarding the current portion.

3. The at least one processor further comprises: - acquiring at least one visual stream (10) corresponding to said continuous data representative of the pose of said at least one eye (4), evaluating said current part of said environment from said at least one visual stream (10), triggering said continuous movement and controlling said dynamic adjustment in particular in function of said at least one visual stream (10), characterized in that it is arranged, during said training phase, to obtain at least one training visual stream (10') corresponding to said training movement sequence data (23) representative of said training movement sequence, and to evaluate said current portion of said training environment and to determine said information in particular in function of said at least one training visual stream (10'), 3. Apparatus (11) according to claim 1 or 2.

4. The device (11) of claim 3, wherein the at least one processor (12) is further configured for conditioning the acquired at least one visual stream (10) and the at least one training visual stream (10') in association with at least one conditioning parameter specific to the mammal being simulated, and the at least one processor (12) is further configured for learning the at least one conditioning parameter during the training phase in function of the at least one training visual stream (10') and the continuous movement, taking into account the evaluation of the current portion of the training environment and the recording of the information relating to the current portion.

5. The at least one processor (12) further comprises: - deriving successive images from said at least one visual stream (10); - configured to control said dynamic adjustment by stabilizing at least a portion of said derived successive images, 5. Apparatus (11) according to claim 3 or 4.

6. The device (11) according to any one of claims 3 to 5, characterized in that the device (11) comprises at least one fourth input adapted to receive at least one simulated visual characteristic, and the at least one processor (12) is further configured for dynamically conditioning the at least one visual stream (10) in function of the at least one simulated visual characteristic.

7. 7. The device (11) according to claim 6, characterized in that the at least one simulated visual characteristic is at least partially related to a physiological behavior associated with the at least one eye (4).

8. 8. Apparatus (11) according to claim 6 or 7, characterized in that said at least one simulated visual characteristic is at least partially related to the visual equipment of said at least one eye (4).

9. The device (11) according to any one of claims 1 to 8, characterized in that it comprises at least one fifth input adapted to receive information relating to a correction value of the stabilization constraint (7), and the at least one processor (12) is configured for replacing information relating to a previous value of the stabilization constraint (7) in the memory (13) with information relating to the correction value.

10. The device (11) according to claim 9, characterized in that the previous and revised values ​​of the stabilization constraints (7) correspond to at least one of a pair of different mammal ages and a pair of different health states.

11. The device (11) according to any one of claims 1 to 10, characterized in that the at least one processor (12) comprises at least one first neural network (18) configured for controlling the dynamic adjustment by means of a feedback loop.

12. The device according to any one of claims 1 to 11, characterized in that the at least one processor (12) comprises at least one second neural network (19) configured for evaluating the current portion of the environment and for recording information about the current portion.

13. the at least one processor (12) is configured to perform at least one of the following operations: evaluating the current portion of the environment (2) by means of a simulated spiking neuron (17), recording the information (8, 8') relating to the current portion, and controlling the dynamic adjustment of the continuous movement (6) of the at least one eye (4), An apparatus (11) according to any one of the preceding claims.

14. A process for simulating the physiological behavior of a mammal in an environment (2) by means of a virtual mammal (1), said virtual mammal (1) comprising: a head (3) that is movable (5) with respect to said environment; at least one eye (4) that is rotationally movable (6) relative to the head (3), The process comprises: - receiving continuous data representative of a pose of said at least one eye (4) relative to said environment (2); - receiving an instruction for at least one movement behavior (9) by said virtual mammal (1) in said environment (2); evaluating, by at least one processor (12), a current portion of said environment (2) from said continuous data using information about said environment (8) stored in at least one memory (13), and if said current portion is not identified in said at least one memory (13), recording information about said current portion (8) in said at least one memory (13); triggering, by said at least one processor, a continuous movement (5, 6) of said head (3) and said at least one eye (4) in function of said at least one locomotor behavior (9), said continuous data and said stored information on said environment (8); and controlling, by the at least one processor, a dynamic adjustment of the continuous movement (6) of the at least one eye (4) to the continuous movement (5) of the head (3) in function of the continuous data by using a stabilization constraint (7) between the movement (6) of the at least one eye (4) and the movement (5) of the head (3), The process further comprises: - receiving, during a training phase, training movement sequence data (23) representative of a training movement sequence of said head (3) and said at least one eye (4) associated with a training environment; learning, by the at least one processor, during the training phase, - triggering, in said training environment, a sequence of movements (5) of said head (3) corresponding to said training movement sequence; - assessing a current portion of the training environment from said training movement sequence data and, if said current portion is not identified in said at least one memory (13), recording information about said current portion (8') in said at least one memory (13); - determining information relating to at least a portion of the stabilization constraints in function of the training movement sequence data and the continuous movement of the head taking into account the evaluating the current portion of the training environment and recording the information relating to the current portion, and recording the information relating to the at least a portion of the stabilization constraints in the at least one memory; and learning at least a portion of the stabilization constraints (7) by As a result, the virtual mammal (1) provides, through the continuously dynamically adjusted current movement of the head (3) and the at least one eye (4) and through evaluating the environment (2), a simulated behavior of the mammal to be simulated when performing the at least one locomotion behavior in the environment. process.

15. A computer program comprising instructions which, when executed by a processor, cause the processor to carry out the process according to claim 14.

Images