A system that simulates the decision-making process in the mammalian brain about visually observed body movements

By introducing a descriptive risk-adverse Bayesian model with forgetting function and a dynamic re-inhibition neural network into the simulation system, the problem of inaccurate simulation of biological motion perception in existing technologies is solved, and more accurate motion perception and decision simulation is achieved, especially in rotation detection.

CN114175046BActive Publication Date: 2024-11-26UNIV DE MONTREAL +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080054700.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-08-05
Filing Date
2020-08-03
Publication Date
2024-11-26
Estimated Expiration
2040-08-03

AI Technical Summary

Technical Problem

Existing results of simulating biological motion perception in non-biological machines are not effective enough to accurately simulate the decision-making process of body motion observed visually in the mammalian brain.

Method used

A descriptive risk-adverse Bayesian simulation model with added forgetting functionality was developed. It combines a feedforward risk-sensitive Bayesian simulation model with a dynamically mutually inhibiting neural network with forgetting functionality to simulate visual information processing in visual receptors and neuronal structures, including local motion direction detection, opposing motion detection, and complex pattern detection. Decision-making is carried out through optical flow data streams.

Benefits of technology

It improves the accuracy of motion perception and decision-making in simulation systems, especially in detecting clockwise and counterclockwise rotation, and can better simulate the decision-making process of human subjects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114175046B_ABST
    Figure CN114175046B_ABST
Patent Text Reader

Abstract

The present invention is a system (1) that simulates a decision-making process in a mammalian brain about motion characteristics related to the body posture of a visually observed body by including a simulated visual path toward an interface of a simulated neuronal structure, the interface at least converting the luminescence information of the observed body into an optical flow data stream, the optical flow data stream conveying information related to the visually observed body and being able to process the information in the simulated neuronal structure, the system being a feedforward system and hierarchically comprising from the visual observation to the decision: the simulated visual path and its interface (3, 3L, 3R); a simulated local motion direction detection neuronal structure (4, 4L, 4R) for detecting motion direction through a receptive field; a simulated opposing motion detection neuronal structure (5, 5L, 5R) for detecting opposing motions at least related to expansion and contraction; a simulated complex pattern detection neuronal structure (6, 6L, 6R) for globally detecting optical flow patterns over the entire visual observation and according to the evolution over the time period of the entire visual observation, the detectable patterns being prototype patterns; and a simulated motion pattern detection neuronal structure (7, 7LR) for detecting motion patterns and providing decisions about motion characteristics. According to the invention, the neurons of the simulated motor pattern detection neuron structure (7, 7LR) comprise a forgetting capacity which is a function of a delay and, for each neuron, of the activity of said neuron.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a detection and decision system that simulates the decision-making process in the mammalian brain regarding visually observed body movements. Background Art

[0002] The powerful ability of humans to recover information about moving living bodies (e.g., identity or activity type) from sparse visual input is known as biological motion perception. Biological motion perception has been the subject of research. Motion perception is an important function of organisms and is used in many activities ranging from basic survival activities to social life in mammals (and especially humans). Biological models of the neuronal structures involved in biological motion perception and their relationships have been developed.

[0003] So far, it has been determined that the biological motion perception function involves the integration of local motion detection and dynamic shape cue recognition. More precisely, this function involves the activation of a "dorsal pathway" dedicated to motion information processing and a "ventral pathway" dedicated to shape recognition in the neuronal system, where the two outputs of these two pathways meet at the superior temporal sulcus (STS).

[0004] In the relevant technical field, the following documents have been identified:

[0005] -(1)YI-ZENG HSIEH et al.: “Development of Home Intelligent Fall DetectionIoT System Based on Feedback Optical Flow Convolutional Neural Network”, IEEE ACCESS, Vol. 6, January 1, 2018 (2018-01-01), pp. 6048-6057. The document discloses a standard network CNN (Convolutional Neural Network).

[0006] - (2) MURUGADOSS R et al.: “Universal approximation of nonlinear system predictions in sigmoid activation functions using artificial neural networks”, 2014 IEEE International Conference on Computational Intelligence and Computational Research, IEEE, December 18, 2014 (2014-12-18), pp. 1-6. This document discloses a modification of the nonlinear kernel on a fully connected known MLP (Multi-layer Perceptron).

[0007] -(3) YIKANG SHEN et al.: “Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks”, ARXIV.0RG, Cornell University Library, 201 Olin Library, Cornell University, Ithaca, New York, 14853. This document discloses a type of RNN (recurrent neural network) called LSTM (long short-term memory), which is well known and uses feedback connections.

[0008] However, the results provided by their simulations in non-biological machines (especially computers), which try to mimic the biological structures involved, are not very efficient and are not close enough to the results provided by the real biological neuronal structures involved in the biological motion perception of living things. Summary of the invention

[0009] The inventors have developed a new descriptive risk-adverse Bayesian simulation model with added forgetting function, which greatly improves the results of motion perception (i.e., detection and decision-making) in the simulation system. Moreover, adding some specific motion detections to the system, especially clockwise and counterclockwise rotation detection, has been shown to further improve the simulation results.

[0010] The proposed system aims to simulate the visual information processing that takes place in the visual receptors (the eye, which may be assisted by ophthalmic devices), and in the neuronal structures downstream of the eye, and in particular some neuronal structures of the visual cortex within the human brain, in order to reproduce the way a specific human subject makes decisions about the content of the visual information he / she receives.

[0011] In the following description, it is assumed that visual information processed and transformed in the eye and the final ophthalmic device involves visual pathways and their interfaces. Downstream of the eye, visual information processed in neurons of a neuronal structure is assumed to involve an optical flow data stream, and the related neuronal structure is generally referred to in the technical field as involving a visual cortical pathway.

[0012] Therefore, a feed-forward risk-sensitive Bayesian simulation model is proposed to mimic biological motion perception. The proposed model is hierarchical and utilizes stored / memorized prototype patterns related to the motion characteristics that the system has to decide. Moreover, to further improve the results, the simulated decision-making neurons are implemented using a dynamic mutual inhibition neural network with forgetting function.

[0013] It is important to note that in this document the word “detector” is used as equivalent to the word “neuron”.

[0014] More specifically, the present invention is a detection and decision system that simulates a decision process in a mammalian brain regarding motion characteristics related to a body posture of a visually observed body by means of a simulated visual pathway comprising an interface toward a simulated neuronal structure, wherein the simulated visual pathway and its interface at least convert luminescence information of the observed body into an optical flow data stream that conveys visual observation information related to the visually observed body and can be processed in the simulated neuronal structure,

[0015] The system is a feed-forward system and the hierarchy from the visual observation to the decision-making includes:

[0016] - the simulated visual pathway and its interfaces, the simulated visual pathway and its interfaces being configured to simulate an eye and a final ophthalmic device on said at least one eye,

[0017] - simulating a local motion direction detection neuron structure for detecting the motion direction through a receptive field and receiving an optical flow data stream from the interface,

[0018] - a simulated opposing motion detection neuronal structure for detecting opposing motions associated with at least expansion and contraction, and receiving an optical flow data stream from said simulated local motion direction detection neuronal structure,

[0019] - a simulated complex pattern detection neuronal structure for detecting optical flow patterns globally throughout a visual observation and as a function of their evolution over a time period of said entire visual observation, and receiving an optical flow data stream from said simulated opponent motion detection neuronal structure, the detectable patterns being prototype patterns, and

[0020] - a simulated motion pattern detection neuron structure for detecting motion patterns and receiving an optical flow data stream from said simulated complex pattern detection neuron structure and providing a decision regarding motion characteristics.

[0021] According to the present invention, the neurons of the simulated motion pattern detection neuron structure all include a forgetting ability, and the forgetting ability is a function of a delay and an activity of the neuron.

[0022] The following additional measures, either alone or in all possible technical combinations, should also be considered:

[0023] -Simulating movement patterns to detect neuronal structures is a neuronal structure that can remember,

[0024] - the action characteristics related to the body posture are selected from at least one of the following: movement direction, movement speed, movement position,

[0025] - the visually observed body is a living body, especially an animal or human body,

[0026] - the visually observed body is a machine, especially a robot,

[0027] -The visually observed body is a virtual body,

[0028] - the optical flow data stream passed between the detection neuron structures carries different types of information depending on which detection neuron structure it comes from,

[0029] - when coming from a simulated oppositional motion detection neuron structure, the optical flow data stream may be a feature vector,

[0030] - The optical flow data stream consists of consecutive frames,

[0031] - prototype patterns are learned by the system during the initial learning phase,

[0032] -Prototype patterns are pre-built in the system,

[0033] - the simulated opposing motion detection neuron structure further detects opposing motions associated with rotation,

[0034] -The internal noise of the system is further simulated in the system,

[0035] - Noise is added to the optical flow data stream output from a simulated complex pattern detection neuron structure,

[0036] - Noise is added to the output of each neuron in the simulated complex pattern detection neuron structure,

[0037] - Noise is added to the optical flow data stream that is input into the simulated motion pattern detection neuron structure,

[0038] The added noise has a Gaussian distribution N(H i (t),Δtδ 2 ), where δ 2 is the variance, and H i (t) is the ideal activity of a complex pattern-detecting neuron in the absence of added internal noise,

[0039] - the simulated visual pathway and its interface are configured to simulate an eye and a final ophthalmic device on the at least one eye,

[0040] - the simulated eye and ophthalmic device have certain characteristics,

[0041] -The characteristics of the eyes are mainly normal vision, myopia, astigmatism...

[0042] - The characteristics of ophthalmic devices are mainly the lens power...

[0043] - visual observation of the body by a camera simulating an eye and generating an optical flow data stream which is further filtered by at least one characteristic function to simulate the characteristics of the eye and the final ophthalmic device,

[0044] -The camera simulating the eye consists of a set of receptive fields,

[0045] - the simulated local motion direction detection neuron structure is configured to detect the motion direction in the two-dimensional space between the receptive fields, and the detectable motion directions are up, down, left, and right,

[0046] - each receptive field has a receptive field comprised between 0.5 degrees and 1.5 degrees,

[0047] -The receptive field is preferably 0.9 degrees,

[0048] - the simulated oppositional motion detection neuron structure is configured to detect expansion, contraction, clockwise rotation and counterclockwise rotation in two-dimensional space,

[0049] -Each neuron of the simulated opposing motion detection neuron structure has a 3.5 degree observation angle,

[0050] - the simulated oppositional motion detection neuron structure is configured to detect expansion, contraction, clockwise rotation and counterclockwise rotation between adjacent receptive fields,

[0051] - the simulated oppositional motion detection neuron structure is configured to detect expansion, contraction, clockwise rotation and counterclockwise rotation between non-adjacent receptive fields,

[0052] -The simulated complex pattern detection neuron structure includes a set of simulated neurons and is configured to detect prototype patterns in a two-dimensional space within a global observation angle with a determined value, the simulated neurons of the simulated complex pattern detection neuron structure being assigned to a plurality of groups in the set, the number of the groups corresponding to the number of possible decisions that the system can provide, each group including simulated neurons that are asymmetrically and laterally connected together within the group.

[0053] - the number of groups is even,

[0054] - For two possible decisions, the number of groups is two,

[0055] - The number of groups is an odd number, with a minimum of three groups,

[0056] - For three possible decisions, the number of groups is three,

[0057] - Each group includes the same number of simulated neurons,

[0058] - when at least one group forms a decision space, the groups include different numbers of simulated neurons,

[0059] - the simulated complex pattern detection neuron structure comprises a set of simulated neurons and is configured to detect a prototype pattern in a two-dimensional space by global observation, the set of simulated neurons of the simulated complex pattern detection neuron structure being divided into two equal groups for two possible decisions about opposite movement directions, each group comprising simulated neurons connected together asymmetrically and laterally within the group,

[0060] -The two opposite directions of movement are left and right,

[0061] -The two opposite directions of movement are up and down,

[0062] - the set of simulated neurons simulating the complex pattern detection neuron structure is divided into four groups for four detectable opposite two-by-two motion directions, namely two opposite directions of left and right and two opposite directions of up and down,

[0063] - The global observation angle of 8 degrees is used to simulate complex pattern detection neuron structures,

[0064] - The global observation angle of the simulated complex pattern detection neuron structure is larger than 8 degrees and corresponds to the entire visual observation,

[0065] - The global observation angle corresponding to the entire visual observation, simulating the complex pattern detection neuron structure,

[0066] - a set of simulated neurons simulating a complex pattern detection neuron structure including eighteen simulated neurons, nine simulated neurons for each detectable opposite direction of motion,

[0067] - in the simulation of complex pattern detection neuronal structures, active neurons excite neurons tuned to predict later patterns and inhibit neurons tuned to past patterns,

[0068] - In the simulation of complex pattern detection neuronal structures, the dynamics of neurons sensitive to optical flow patterns correspond to the following regularity:

[0069]

[0070] Where i is the i-th frame of the optical flow data stream, the optical flow rises from the i-1th frame to the i-th frame, Hi(t) is the activity of the neuron, τ OFP = 150ms, w(m) is the weight kernel, f(H) is the step threshold function, and Gi(t) is the instantaneous feedforward input of the neuron,

[0071] - In the simulated complex pattern detection neuron structure, the implemented descriptive risk-averse Bayesian classifier is based on an average risk minimization approach with different penalty levels for different errors, where a classification error of the current frame as one of the later feature vectors has a lower severity level than a classification error regarding an older template, which occurs in the simulated opponent motion detection neuron structure output and involves spatial R i The feature vector u of is incorrectly classified as class ω k When i≠k, and in the case of misclassification, the loss term λ ki Assign to the wrong classification, and set the loss term λ to 0 in case of correct classification ki Assigned to the correct classification, and formed including the element λ ki The loss matrix, whose elements constitute the loss matrix for classifying as class ω k The true state is the penalty level of i, which simulates the feature vector received by the complex pattern detection neuron structure.

[0072] In the case where the set of simulated neurons simulating the complex pattern detection neuron structure comprises eighteen simulated neurons (nine simulated neurons for each detectable opposite direction of motion), eighteen classes are considered (nine classes for each of the two opposite directions of motion), minimizing the average risk according to the following formula:

[0073] u∈R i if Then ll =0,

[0074] Among them, ω i (i=1…18) is the class, R j (j=1…18) when it has the lowest weighted sum and is classified as class ω i When the feature space R i The region / subspace of , and where p(Pu|ω k ) is in a given class ω k The likelihood of the feature vector in the case of , and p(ω k ) is a class ω k The prior probability of each space R i The likelihood of the eigenvector of follows a Gaussian distribution N(μ i ,Σ i ), where μi is the mean vector and Σi is the covariance matrix, and for each class ω k The prior probability p(ω k ),in, So that for each sequential frame, as long as u(t) belongs to the feature space R i , the detector tuned to class ωi receives a positive non-zero feedforward input G i (t), and causing neurons with matching templates to inhibit other neurons that mimic the motion pattern detection neuron structure,

[0075] - the simulated motion pattern detection neuron structure is configured as a mutually inhibiting simulated neuron network, wherein the simulated neuron with the highest excitatory input inhibits the activities of other simulated neurons in a non-linear and interactive operation, the activities of the other simulated neurons not exceeding their defined thresholds,

[0076] - The simulated neurons that simulate the mutual inhibition of the motor pattern detection neuron structure are simulated according to the following relationship:

[0077]

[0078]

[0079] Wherein, T is the activity of the first neuron that is excited by the optical flow data stream received from the simulated complex pattern detection neuron structure, and D is the activity of the remaining neurons of the network, τ is a time constant, and S() is a modified Michaelis function according to the following formula (other similar functions such as cumulative Gaussian function or Weibull function can also be used):

[0080]

[0081] where M is the maximum information threshold of excitatory-inhibitory activity, and σ usually marks the information threshold point where the function S(P) reaches half of its maximum value, PT and P D are the information thresholds available to T-type and D-type neurons, respectively, and are:

[0082] P T (D)=E T -kND

[0083] P D (D, T) = E D -k(N-1)D-kT

[0084] where N is the number of neurons and the constant k is the inhibitory feedback gain, E T and E D represents the optical flow data stream received from the simulated complex pattern detection neuron structure,

[0085] When the mutually inhibiting simulated neurons of the network receive a negative input in the optical flow data stream from the simulated complex pattern detection neuron structure, then the mutually inhibiting simulated neurons are simulated according to the following relationship:

[0086]

[0087]

[0088] - The simulated neurons that simulate the mutual inhibition of the motor pattern detection neuron structure have additional inputs, through which the forgetting ability is implemented, and the additional inputs are for the neurons that are excited first according to Dis T =u(t-τ a )*[TS(P T (D))-k Dis T] to receive data, and for other neurons according to Dis D =u(t-τ a )*[DS(P D (D, T))-k Dis D] to receive data, and wherein T is the activity of the first neuron that is excited by the optical flow data stream received from the simulated complex pattern detection neuron structure, and D is the activity of the remaining neurons of the network, u() is a unit step function, τ a is a time constant, and S() is a modified Mie function or any other function having the same mathematical symmetry as the Mie function (e.g., a cumulative Gaussian function or a Weibull function), and k Dis is the weighting coefficient,

[0089] - said system comprises at least one programmable computer implementing said neuronal architecture,

[0090] - said at least one programmable computer comprises an input-output interface allowing the input of parameter values ​​modifying at least the delay associated with said forgetting capacity,

[0091] - the system further comprises a learning function implemented with a risk-sensitive Q-learning algorithm,

[0092] - The learning function is implemented in the simulated complex pattern detection neuron structure. DETAILED DESCRIPTION

[0093] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0094] - Figure 1 The structure of the system according to the invention is schematically shown in the case of monocular observation,

[0095] - Figure 2 schematically illustrates the structure of the system according to the invention in the case of binocular observation, and

[0096] - Figure 3 is an example of the learning steps of the Q-learning algorithm implemented in the third layer of the system.

[0097] The system of the present invention will now be described using an exemplary embodiment, and then the application of the system in a soccer game will be presented. More specifically, the exemplary embodiment is a system capable of discerning the direction of the ball from a complex biological motion soccer shot visual stimulus.

[0098] More generally, the system of the present invention for simulating decision-making processes is based on neural mimicry, and the system simulates the neuronal operation of the mammalian brain, in particular the "dorsal pathway" dedicated to motion information processing, which processes "optical flow" containing visual information. The decisions provided by the system are related to motion characteristics (e.g., presence of motion, direction of motion) associated with visually observed body postures. In this sense, the system, while acting as a simulation system, also behaves as a real biological motion detection and decision-making organ.

[0099] In fact, for a system without internal adaptability (i.e., without being able to adjust its neurons to detect and determine new characteristics on its own), the characteristics that the system is able to determine are the characteristics that the system is configured to detect and determine. In all cases, the result / output of the system is a decision choice related to the characteristic. It should be noted that the system of the present invention has a certain learning ability, in the sense that the neuronal structure operates according to functional parameters that can be learned through a training procedure in order to reproduce in particular the decision-making style of a specific human subject. In a simpler operation, the system is directly provided with already predefined functional parameters. We will see that these parameters can be, for example, the noise level, the degree of inhibition between neurons, the intrinsic delays of neurons.

[0100] In principle, the simulated detection and system is a descriptive risk-averse Bayesian decision-making system. The system is based first on the assumption that the visual system imitated by the simulated system stores prototype patterns during perception / cognition, and second on a hierarchical feedforward model consisting of stacked layers, each with a specific detection function.

[0101] Additionally, the last layer of the hierarchy (i.e., the motion pattern detector layer, and its simulated decision making neurons) is based on a dynamical model of a network of neurons with mutual inhibition. However, this last model has been refined by an additional function, namely the forgetting function. It is the activity of these motion pattern detectors that constitutes the decision response, or more generally, the behavior of the biological motion simulation detection and decision making system.

[0102] Therefore, the neuron model used in the system of the present invention is based on three main assumptions:

[0103] - The simulated dorsal stream consists of a hierarchy of neuronal detectors (ie, neurons) configured to extract optical flow features.

[0104] -The model uses a feed-forward architecture between neuron structure layers.

[0105] -The system stores prototype patterns and uses them for perception and recognition.

[0106] about Figure 1 and Figure 2 , the simulated neuronal hierarchy of the dorsal stream is a feed-forward architecture comprising the following neuronal layers operating sequentially / serially in the simulated neuronal structure:

[0107] - Local motion energy detector 8,

[0108] - Opposing motion detector 9,

[0109] - a complex pattern detector arranged as two detector groups 10, 11,

[0110] - Movement pattern detectors 12,13.

[0111] These simulated neuronal structures are linked in a hierarchical manner and information is passed between them in the form of optical flow data streams from the output of the lower neuronal structure in the feed-forward architecture / hierarchy to the input of the next neuronal structure.

[0112] Depending on the actual implementation of computations to simulate neurons of these neuronal structures, optical flow dataflow can be implemented in a variety of ways, from single messages between specific entities simulating neurons to global message passing. In all cases, the underlying principle behind this is neuronal computation, where the output of a detector / neuron is computed based on its input, mimicking a biological neuronal cell.

[0113] Neuronal structures and neuronal hierarchies can be implemented in a variety of ways, from a fully virtual system that is fully computed to using hardware elements that simulate neurons, interconnected, for example as an ASIC or a reprogrammable gate array (where groups of gates are arranged to simulate neurons).

[0114] It must be noted that in the exemplary embodiment, since a general purpose computer is used as a digital system, the embodiment is based on sampled data, and the evolving visual data processed within the system is sampled into a series of still images or frames. However, in the case where an analog based system is used and the visual data is continuously available, a non-sampled embodiment can be used.

[0115] The system receives a sequence or video of visual stimuli, typically a series of frames or still images, which are processed sequentially one after the other and each layer outputs an optical flow data stream carrying different types of information, depending on the detection function of the layer. The sequence of visual stimuli can be provided directly to the system, or it can be the output of a device simulating an eye, in particular the output of a digital camera that performs some processing on the data provided by the camera. By means of such simulated visual paths and interfaces 3 in the system, it is then possible to simulate an eye in a perfect state, or alternatively, an eye with a certain disease (or diseases) or defect.

[0116] The system according to the invention was challenged to detect the direction of a ball from the perspective of a simulated subject with a simulated decision-making brain part based on the system according to the invention under high-complexity visual stimuli (i.e., a soccer shot on the ball). Therefore, the described exemplary embodiments are constructed based on this goal / objective.

[0117] Local Motion Energy Detector

[0118] The local motion energy detector is implemented in an analog local motion direction detection neuronal structure 4, 4L, 4R, which forms the lower first layer in the neuronal hierarchy of the dorsal stream.

[0119] These local motion energy detectors are sensitive to different motion directions and have receptive fields that are smaller than the viewing angle of the simulated opposing motion detection neuron structure, where each receptive field typically has an angle of about 0.9 degrees. In the current exemplary embodiment, the receptive fields are sensitive to four different directions: left and right, up and down between adjacent receptive fields, and the corresponding detectors have been deployed in a 36×31 grid assembly.

[0120] In more advanced embodiments, the detection may be extended to diagonal direction(s).

[0121] The neuronal structure uses two consecutive frames in the visual stimulus sequence for detection: the local motion energy detector detects the change between two consecutive frames, and the optical flow data stream output by the simulated local motion direction detection neuronal structure carries information about the possible motion directions for each receptive field.

[0122] Implementing max pooling in a simulated local motion direction detection neuron architecture.

[0123] If desired, a detailed explanation of the implementation of the simulated motion direction detection neuronal structure can be found in Casile A, Giese MA., "Critical features for the recognition of biological motion", Journal of Vision. 2005; 5(4): 6-, and also in Smith, AT, & Snowden, RJ (1994), "Visual detection of motion"; Academic Press.

[0124] Opposition motion detector

[0125] The opponent motion detector 9 is implemented in an analog opponent motion detection neuronal structure 5, 5L, 5R, which forms the second layer in the neuronal hierarchy of the dorsal stream.

[0126] These detectors are sensitive to opposing motions such as expansion, contraction, and rotation.

[0127] For example, a neuron dedicated to vertical contraction detection is activated by such opposing motion occurring in two adjacent subfields located within its viewing angle, which are confined within the receptive field corresponding to the local motion energy detector.

[0128] In an exemplary embodiment, the viewing angle of each opposing motion detector is approximately 3.5 degrees.

[0129] The opponent motion detector aggregates the responses of local motion energy detectors with the same direction preference in one subfield, i.e., the optical flow data stream output by the structure that simulates the local motion direction detection neurons.

[0130] In the case of vertical pinching this would be rightward motion in the left subfield, while detector responses from adjacent subfields are aggregated in the same way, but with the preference reversed (in the case of vertical pinching the preference would be leftward motion in the right subfield).

[0131] In an exemplary embodiment, the detectors sensitive to the opposing motions of vertical and horizontal expansion and contraction are 100 contraction and expansion detectors arranged as 5×5 components, 25 detectors for horizontal expansion, 25 detectors for horizontal contraction, 25 detectors for vertical expansion, and 25 detectors for vertical contraction, and these detectors are therefore capable of producing detections of 100 features associated with these opposing motions.

[0132] In addition to horizontal and vertical contraction and expansion detection, clockwise rotation detection and counterclockwise rotation detection are implemented that can produce detection of 20 features associated with each rotation orientation (clockwise or counterclockwise), each detection having 20 detectors arranged in 5×4 components. Therefore, the clockwise rotation detector and the counterclockwise rotation detector manage 20 receptive fields, which correspond to the receptive fields of the local motion energy detectors of each clockwise detection and counterclockwise detection, covering a 36×31 grid component that simulates the local motion direction detection neuronal structure. Each rotation receptive field is connected to 4 connected and overlapping sub-fields, allowing the rotation detector to also use a maximum pooling strategy to pick up the highest rotation activity. Each sub-field is associated with a 14×14 local motion energy detector that simulates the local motion direction detection neuronal structure, which constitutes a 5×4 receptive field arrangement for simulating clockwise or counterclockwise rotation in the opposing motion detection neuronal structure.

[0133] Now the structure for the rotation detection mechanism is described. Each rotation detector (clockwise or counterclockwise) is connected to four subfields: upper left, upper right, lower left and lower right subfields. Because the local motion energy detector detects these four basic motion directions: right, left, up and down, a clockwise rotation can only be detected by the neuron under the following two direction conditions:

[0134] - the upper left subfield is upper, the upper right subfield is right, the lower left subfield is left, and the lower right subfield is lower,

[0135] - The upper left subfield is right, the upper right subfield is bottom, the lower left subfield is top, and the lower right subfield is left.

[0136] The two conditions for detecting counterclockwise rotation can be easily inferred from the conditions for this rotation.

[0137] Thus, the output of the simulated oppositional motion detection neuron structure is able to detect 140 features associated with expansion, contraction, and rotation corresponding to oppositional motion, which are output in the optical flow data stream that is sent to the next layer.

[0138] Complex pattern detector

[0139] The complex pattern detectors are arranged in groups 10, 11 and implemented in an analog complex pattern detection neuronal structure 6, 6L, 6R forming the third layer in the neuronal hierarchy of the dorsal stream.

[0140] A pattern is a series of time-varying features that can be detected by the simulated oppositional motion detection neuronal structure and that correspond to a specific decision that the system can reach. A pattern is a learned template and it is a statistical frame. In the current exemplary embodiment, there are 18 statistical frames (two times nine, for both left and right directions), and each neuron / detector of the simulated complex pattern detection neuronal structure 6, 6L, 6R is configured with one of these statistical frames. In an exemplary embodiment, these statistical frames are pre-set in each of these neurons and there is no learning process. However, in more advanced embodiments, an initial learning phase can be implemented and performed. Therefore, in an exemplary embodiment, these patterns are pre-established / pre-set based on possible decisions that the system is configured to calculate regarding visualized and processed movements related to body posture.

[0141] The complex pattern detection neuron architecture consists of neurons that are able to detect / discriminate a series of temporally complex optical flow patterns over time. These neurons then also respond to the temporal order of the inputs they receive. This means that if the order is incorrect, the risk (see Bayesian risk assessment scheme explained below) will be significantly higher, so the neuron remains silent.

[0142] The complex pattern detectors are arranged as laterally coupled neuronal networks, where asymmetric connections allow active neurons at a certain moment to excite neurons tuned to possible (multiple) later optical flow patterns, and inhibit the remaining detectors from encrypting past / older optical flow patterns. If necessary, you can consult the reference: Mineiro, P. and Zipser, DJNC (1998), "Analysis of direction selectivity arising from recurrent cortical interactions" 10(2), 353-371.

[0143] In this way, the hypothesized dynamics of a complex pattern detector sensitive to the i-th frame (i.e., the optical flow from the i-1th frame and the i-th frame in the frame sequence) is as follows:

[0144]

[0145] Where t is the time, m corresponds to the number of neurons in the third layer, and H i (t) is the activity of the ith neuron, τ OFP = 150ms is the time constant of the complex pattern detection dynamics, w(m) is the asymmetric weight kernel, f(H) is the step threshold function, and G i (t) is the instantaneous feedforward input of the neuron and comes from the second layer 5, 5L, 5R. More precisely, the input of the simulated complex pattern detection neuron structure 6, 6L, 6R (which comes from the simulated oppositional motion detection neuron structure 5, 5L, 5R) is subjected to a Bayesian risk assessment scheme, and the result constitutes Gi(t). The inhibitory part of these formulas is the summation term.

[0146] Since the feed-forward input to each optical flow neuron in the third layer 6, 6L, 6R is derived via a Bayesian risk assessment scheme, this represents a memory that simulates a complex pattern detection neuronal structure 6, 6L, 6R.

[0147] The dynamics of the complex pattern detector of the third layer 6, 6L, 6R shown above are solved using Euler's method.

[0148] As mentioned earlier, one of the basic assumptions about the model is the prototype matching performed by the neurons. This prototype matching is nothing more than the result of the template matching process that constitutes the aforementioned feedforward input. The information that the template corresponds to is based on prior knowledge or previous learning of possible patterns.

[0149] Attempts to develop laterally coupled neuronal networks for pattern detection are described in Casile, A. and Giese, M.A. (2005), “Critical features for the recognition of biological motion,” Journal of Vision, 5(4), 6-6, doi: 10.1167 / 5.4.6 and also in Giese, M.A. and Poggio, T. (2003), “Neural mechanisms for the recognition of biological movements,” Nat Rev Neurosci, 4(3), 179-192.

[0150] To generate the instantaneous feedforward input G i (t), the exemplary embodiment uses a smaller number of neurons and a different strategy than the developments described in the two documents just listed above. In these documents, radial basis functions are used for Gi(t). Moreover, radial basis functions do not work at all for the type of stimuli processed in current systems.

[0151] In an exemplary embodiment, the feedforward input G i (t) is considered as a product of a multi-class Bayesian classification scheme. Instead of using the classic classification error minimization, we choose to use an approach that minimizes the average risk and includes different meanings for different types of errors. This has been shown to be more efficient. More precisely, the severity of misclassifying the current frame as one of the later feature vectors must be lower than the severity of misclassification associated with the past / older feature vector / template.

[0152] The logic behind this can be explained with the goalkeeper example; it means that if the goalkeeper decides that the frame observed in the scene belongs to one of the future states of the sequence, the chances of saving the ball will be less affected than if the encoded template classified the scene as past / previous, remember that the template is an array of 140 elements, so it encodes a frame of 300×400 pixels.

[0153] In an exemplary embodiment, with respect to the motion characteristics, all the system must determine is whether the direction of the ball resulting from the movement of the soccer player is to the left or to the right.

[0154] To simplify the process, the angle of deviation of the shot relative to the center of the visual observation scene in which the system sees the body of the football player is considered. Still for simplicity, nine consecutive time periods of each stimulus sequence (i.e., penalty sequence) have been considered for defining these patterns with respect to their evolution over time.

[0155] In an exemplary embodiment, each stimulus sequence / video with a duration of 4.5 s is sampled and consists of 90 frames, and the simulated complex pattern detection neuron structure receives a feature vector of 140 elements at its input for every two consecutive frames. The sampling is based on time points with a period of 500 milliseconds. The simulated complex pattern detection neuron structure includes eighteen detectors / neurons, of which groups 10 ( Figure 1 ) of nine detectors for the right direction, and group 11 ( Figure 1 ) for the left direction, and each detector is sensitive to ten consecutive frames out of the 90 frames of stimulation. The template stored in each simulated neuron is a statistical pattern representing all of these 10 frames and these ten frames at 20 different angles, in addition to the parallel connections between neurons that control the neurons to maintain the sequence.

[0156] More generally, the number of neurons in the complex pattern detection neuron structure is a function of the number of frames that each neuron is dedicated to processing (10 frames in the exemplary embodiment), the length of the stimulus (90 frames in the exemplary embodiment), and the number of possible decisions (2 in the exemplary embodiment). Of course, in other embodiments, these values ​​may vary.

[0157] For example, in an exemplary embodiment, a neuron Frames 1 to 10 are selective for left-hand shots, while neurons Frames 41 to 50 are selective for right-side shots. Each of these 18 neurons contains an internal generative model (u|ω) assumed to be of Gaussian form N(uk,Σk) k ), k = 1, ..., 18. The mean uk and covariance matrix Σk of each template are calculated using feature vectors derived from multiple stimuli with different degrees of deviation from 10 frames. Therefore, each template and therefore each statistical frame is a Gaussian function with a mean vector and a covariance matrix.

[0158] For example, the neuron is trained using the feature vectors from frames 61 to 70 of the shot The frames were offset from the observer’s right hand side by 7° to 20°. In short, the neurons All should be selective for frames 61 to 70.

[0159] These parameters result in a classification problem with eighteen classes, nine for right-facing balls and nine for left-facing balls. This also means that the simulated complex pattern detection neuron structure includes 18 detectors / neurons, 9 for right-facing and 9 for left-facing.

[0160] Each class represents a specific period in the stimulus sequence. Each period corresponds to 10 consecutive frames in a stimulus sequence with 90 frames. For example, class 1 associated with ball going right means that we are in the first period in the stimulus sequence and this corresponds to the first ten frames. Class 3 associated with ball going left is a good detection when the stimulus sequence arrives somewhere between frame 31 to frame 40 and has this ball orientation.

[0161] Therefore, the relevant classification problem is a problem with eighteen classes, ω i , i=1,…,18, where R j (j=1,…,18) constitutes the feature space R i Of course, this can be applied to any number of classes (neurons) in the third layer.

[0162] When it belongs to space R i The feature vector u is misclassified as class ω k However, an error occurs when i≠k, and therefore, the loss term λ ki will be assigned to the wrong decision. In this way, a loss matrix can be formed whose elements λ ki Constructs the action k (here: classified as class ω) when the true state is i (the feature vector fed to the third layer) k ) punishment.

[0163] It can be seen that the average risk is minimized when:

[0164] u∈R i ,if Then ll =0,

[0165] This shows that when u has the lowest weighted sum and is classified as class ω i When it comes from space R i In this inequality, p(u|ω k ) is in a given class ω k The likelihood of the eigenvector in the case of , and p(ω k ) is a class ω k In the implemented model, it is assumed that each space R i The likelihood of the eigenvector of follows a Gaussian distribution (μ i ,Σ i), where μ i is the mean vector, and Σ i is the covariance matrix. And, for each class ω k The prior p(ω k ),and

[0166] In this way, as long as u(t) belongs to the feature space R i , tuned to class ω i The detector / neuron receives a positive non-zero feed-forward input G at each time step / frame i (t).

[0167] This can be described at the cellular level in the following way: when an input matches the preset template of one neuron labeled i, all other neurons of different classes treat this as a sizable weighted amount added to their risk sum, and the loss term λ ii = 0, the matching neuron labeled i is exempted from adding this large signal to its risk sum. In other words, neurons with a matching template inhibit the responses of other neurons.

[0168] Motion pattern detector

[0169] The motion pattern detectors 12, 13 are implemented in an analog motion pattern detection neuronal structure 7, 7LR, which forms the fourth layer in the neuronal hierarchy of the dorsal stream and provides the final decision regarding the motion characteristic(s).

[0170] Therefore, the recognition of complete biological motion patterns occurs in the motion pattern detectors / neurons that make up the top fourth layer of the model.

[0171] In an exemplary embodiment, the complete biological action is limited to two possible decisions: distinguishing between a stimulus to shoot left and a stimulus to shoot right, and only two simulated neurons are implemented. These two neurons are arranged according to a robust mutual inhibition model with adaptation and forgetting capabilities. This is a nonlinear differential system and has been solved using a 4th order Runge-Kutta method.

[0172] More generally, the sum of the activities of the complex pattern detectors implemented in the simulated complex pattern detection neuronal structure and belonging to one particular action corresponding to one of the possible decisions is used as input to the motor pattern detector associated with that action.

[0173] In case the number of activity decisions that the system has to decide is not two but for example three, then three neurons are implemented in the fourth layer and the third layer neurons are initially trained with these three activities to be decided by the system.

[0174] As mentioned at the beginning, this is a non-linear, excitatory, inhibitory neuronal network that is implemented to simulate the motion pattern detector of the fourth layer.

[0175] To do this, a nonlinear, excitatory and inhibitory network is used to simulate these motor pattern detectors. The operation of this network is known as mutual inhibition or global inhibition, where the detectors with the highest excitatory inputs inhibit the activity of those detectors whose activity does not exceed their threshold in a nonlinear and reciprocal manner.

[0176] In addition, a forgetting function was added to the operation of the fourth layer of the neural network. Due to this forgetting function, the system can more accurately simulate the reaction time that exists when a real biological brain makes decisions.

[0177] In the current example, the forgetting function is implemented only in the last two decision neurons of the fourth layer.

[0178] The forgetting function is based on neuronal adaptation, which refers to the decrease in the responsiveness of neurons exposed to constant stimulation over time. For example, in the visual system, if there is no fixational eye movement, the visual perception of an image or motion will gradually weaken or disappear. Motion pattern detector neurons, which are decision-making neurons, are no exception and should not remain activated permanently. The forgetting function is used to implement the transient nature of neuronal activation in the system.

[0179] In the exemplary embodiment, it must be noted that in order to simplify the implementation of the system, the forgetting function is implemented in the fourth layer in the simulated motion pattern detection neuron structure.

[0180] The mutual inhibition model has the following dynamics and explains the responses of decision-making neurons:

[0181] τdT / dt=-T+S(P T (D)

[0182] τdD / dt=-D+S(P D (D, T)

[0183] Wherein, T is the activity of the neuron that is excited first, and D represents the activity of other neurons in layer 4 7, 7LR, τ is a time constant, and S(P) is a modified Michaelis function that has been shown to have advantages in the design of excitatory-inhibitory network models. In other embodiments, another function other than the Michaelis function may be used, for example, a behavioral function such as a cumulative Gaussian function. In such other embodiments, any other function having the same mathematical symmetry as the Michaelis function may be used, for example, a cumulative Gaussian function or a Weibull function.

[0184] In addition, P T and P Dare the information thresholds available to T-type and D-type neurons, respectively: When the information threshold is negative, as a result, the lateral connections of neurons in layer 4 7, 7LR are closed and the robust mutual inhibition model sets the negative output of the neuron to zero. More precisely:

[0185] PT(D)=E T -k inh ND

[0186] P(D, T) = E D -k inh (N-1)D-kT

[0187] Where N is the number of neurons and the constant k inh is the inhibitory feedback gain. In addition, E T and E D Represents the external input generated from the previous layer (the third layer).

[0188] The modified Michaelis function is:

[0189]

[0190] where M is the maximum information threshold for excitatory-inhibitory activity, and σ sets the point at which S(P) reaches its half-maximum value.

[0191] The number of equations to be solved depends on how many decision-making neurons are involved in the process. For example, for a decision-making agent, in order to choose one choice from N choices, it is necessary to solve one equation for P(D) and N-1 equations for P(D). D (D, T) equation.

[0192] In the exemplary embodiment, deciding between left and right, N=2, and thus the fourth layer is implemented with two simulated neurons.

[0193] The forgetting function is implemented in the fourth layer by making the following modification to the mutual inhibition model, which introduces a time-dependent term as an input to the dynamics of both T and D neurons:

[0194] Dis T =(t-τ a )*[TS(P T (D))-k Dis T]

[0195] Dis D =(t-τ a )*[DS(P D (D, T))-k Dis D]

[0196] Where u() is the unit step function, τa Mark the time point when adaptation / forgetting begins, and k Dis is the weighting coefficient. At time τ a At , the forget input of the differential equation is turned on and the dynamics of each neuron weakens to a simple exponential decrease, bringing the neuron out of its excited state.

[0197] Preferably, the implemented mutual inhibition model is also modified, where, in order to reduce the sensitivity level, the fourth layer ignores negative changes instead of exponentially decaying the activity of the neuron according to a linear first-order dynamic. More precisely, when a neuron is disconnected due to a negative information threshold, the neuron activity will be as follows:

[0198]

[0199]

[0200] noise

[0201] The system described so far with four layers can be used in this way. However, in order to better model the uncertainty in the decision making process, it may be preferable to add a certain level of noise.

[0202] In this exemplary embodiment, in order to simplify the implementation of the system, noise is received in the optical flow data stream at the input of the fourth layer. However, noise can also be introduced in one or more other layers of the hierarchical system. Moreover, very low noise levels may also have a beneficial effect on the operation of the neural network.

[0203] Therefore, to model the uncertainty in the decision-making process, we assume that the output of each complex pattern detector (layer 3) is drawn from a Gaussian distribution N(H i (t), Δtδ 2 ), where H i (t) is the variance added in the absence of 2 Average ideal activity of a complex pattern detector in the presence of internal noise.

[0204] Simulating visual paths and interfaces

[0205] It may be advantageous to apply a system for simulating decision-making processes to visually observed / perceived data and also simulate the organ responsible for vision (i.e., the eye). Figure 1 ), 3R and 3L( Figure 2 )The simulated eye will observe scene 2 ( Figure 1The simulated eye converts the luminous information (represented as a series of frames in the image) into an optical flow data stream, which conveys information related to the visually observed scene and can be processed by a four-layer neuron structure. Therefore, the simulated eye is also an interface between the observed scene and the optical flow data stream. This is why the simulated eye is called a simulated visual pathway and interface.

[0206] Simulated eye can be implemented as simulating ordinary eye, perfect eye or eye with some defects.And, can use additional ophthalmic device (for example, lens or goggles) to simulate it.

[0207] The simulated visual path and its interface can be a combination of a camera, a computing device and a visual sensor such as a camera, or simply implemented as a computing device. The computing device allows the optical flow data stream to be pre-processed before being sent to the first detection layer for processing. For example, the pre-processing can be scene blur, scene deformation for simulating myopia or other defects, scene deformation for simulating the action of ophthalmic devices (e.g., lenses or goggles).

[0208] In the present application, the scene comprises an observable body which is moving to create a body gesture analyzed by the system. If the body is preferably a mammalian body and in particular a human body, the invention may be applied to any observable element which may have parts which move in relation to other parts, not necessarily a living being.

[0209] Training and Testing

[0210] Already used on computer The simulation has been implemented and has been used Framework performed the data and statistical analyses.

[0211] For system testing, a simplified body was used, the observation of which was limited to a certain number of bright spots. Figure 1 The reason why the frame sequence of the observed scene 2 includes bright spots moving from frame to frame.

[0212] However, the system can be applied to scenes in which the body has a regular appearance, but this necessarily requires greater complexity of the layers, and in particular more neurons in layer 3 6, 6L, 6R and layer 4 7, 7LR.

[0213] More precisely, to test the system, penalty kick soccer penalty shootout videos recorded from real human soccer players were used. There were no dots on the ball (the ball or any representation of it was not present in the stimulus). The penalty shootout outcome to be judged was the direction of the ball.

[0214] Penalty soccer penalty kick videos were previously used to collect psychophysical data from real human observers (i.e., subjects), especially regarding the decision about the direction of the ball (left or right). In addition to the decision, the collected psychophysical data were especially used to describe the classical thresholds and slopes of the sensory task. These psychophysical data were used for system testing and cross-validation between real humans and the system.

[0215] The stimuli in the video included 15 points representing the head and the main joints of the human body: shoulders, hips, elbows, wrists, knees, and ankles. Figure 2 Schematically represented above, wherein the body of the football player in the observation scene 2 includes points 14 (the movement of these points is represented by arrows). Each stimulus consists of 90 frames with a duration of 4.5 seconds. By rotating the original stimulus around the Z axis, stimuli of penalty football shots to the left and right at different angles can be created.

[0216] In a psychophysical study, 35 human subjects were exposed to stimuli that were angled 2°, 4°, 8°, and 15° away from the left or right hand side of the observer / subject.

[0217] For training and cross-validation of the system, the data used included all shot angles between 1° and 20°, which is the range of angles when a penalty kick is shot from the perspective of the goal.

[0218] The system has been cross-validated using a k-fold cross-validation procedure (k=5).

[0219] More precisely, the model was trained in two stages: first, cross-validation on 1 to 20 degrees to make sure the algorithm was validated, and second, model training on 7 to 20 degrees, which is more meaningful in a natural environment. More precisely, this second stage was performed for angles of 2°, 4°, 8° and 15° to reproduce psychophysical study conditions.

[0220] In the psychophysical study, a forced choice paradigm task has been designed that relies solely on biological motion signals to decide the direction of the ball (left or right). Each of the 35 human subjects was exposed to a total of 1080 randomized stimulus sequences of left and right shots with deviation angles of 2°, 4°, 8° and 15° (120 for each angle on each side). Thus, for each human subject, a psychophysical function was determined that relates human behavior to the angular deviation.

[0221] In this system, in order to be able to reproduce the performance of real humans / subjects, three variables of the system were adjusted to mimic the behavior of 35 subjects in a psychophysics study. These variables are therefore functional parameters of the system and its neuronal structure.

[0222] As in the psychophysical study, for each angle and each side the system has been exposed to the same stimulus 120 times to generate the error percentages. Furthermore, this has been repeated 30 times and the same psychophysical parameters as those of the subjects have been evaluated on the system based on the simulation results to enable a comparison between the subjects and the system.

[0223] The four variables used to tailor the system to the behavior of each of these 35 subjects were:

[0224] - The standard deviation δ of the internal noise received at the input of the motion pattern detector implemented in the analog motion pattern detection neuron structure.

[0225] - The time constant τ of the motion pattern detector implemented in the analog motion pattern detecting neuron structure.

[0226] - The inhibitory feedback gain k of the motion pattern detector implemented in the simulated motion pattern detection neuron structure inh .

[0227] -Embedded into τ a Forgetting in parameters.

[0228] After training, the system performed significantly better than previously known systems that did not have the ability to forget and did not have the ability to detect rotation. A 5-fold cross validation of the system resulted in an average success rate of 87.5%, with the lowest success rate being 62.5% on the first fold and 100% on the last two folds.

[0229] Even though the addition of internal noise levels results in higher angle thresholds, flatter slopes, faster reaction times, or worse overall system performance, the system performance exhibits remarkable robustness in the presence of a wide range of applied internal noise δ.

[0230] Besides noise, which is an important tuning parameter of the system, the degree of inhibition occurring between these two decision-making neurons (denoted by k) and their intrinsic delay (denoted by τ) are shown to be key factors in bringing the model to different subject behaviors.

[0231] For a wide range of variables k, τ, and δ, the system behaves as if it were a constant τ. a = 1.22 seconds to run, and the angle threshold, slope and reaction time are calculated accordingly. Of course, the value of τ can also be adjusted.

[0232] Adding rotation detection resulted in better performance for all calculated configurations of the system without exception, meaning lower angle thresholds and steeper slopes, but with no significant change in reaction time. This addition of rotation detection even allowed simulations to be performed on one of the subjects, which was not possible previously without rotation detection.

[0233] We now present the contribution of the variables to the decision-making behavior of the system obtained from the tests.

[0234] An increase in the dynamical time constant τ of a neuron always leads to better performance, meaning a lower angle threshold and a steeper slope.

[0235] The increase of the inhibitory gain k shows different characteristics (depending on the value of k), but it always leads to a higher reaction time. Initially, the increase of the inhibitory gain k leads to a deterioration of the performance (higher threshold and flatter slope), but this starts to improve after k exceeds the value of 8. However, for δ = 0.030, this phenomenon seems less obvious. At this noise level, the angle threshold does not decrease, but increases at a lower rate. Finally, it must be noted that the slope shows a new trend of becoming steeper, which can only mean that the highest accuracy is achieved at lower angular deviations.

[0236] The improvement in performance is not without cost. Analysis of the activity of the motor pattern detectors showed that when the inhibitory gain k is very large, it takes much longer for the winning detector / neuron to reach the peak of its activity, and this can be interpreted as resulting in longer processing times. Thus, the increase in processing time that occurs under large inhibitory gain, affecting the excitation / inhibition ratio of the motor pattern neuron system, can be interpreted as a remarkable agreement between this system and current findings in humans.

[0237] Results of the subjects and simulation results

[0238] By adjusting the variables mentioned above: internal noise δ, mutual inhibition time constant τ and inhibitory gain k, the system can simulate the behavior of 35 subjects (ie, their psychophysical functions) with very good accuracy.

[0239] Correlation analysis showed that there was a significant positive correlation between the experimental angle threshold and the simulated angle threshold, where the Spearman correlation coefficient was rs = 0.991, p value = 7.08E-31 (p < 0.001), and there was another significant positive correlation between the simulated slope value and the experimental slope value, where the Spearman correlation coefficient was rs = 0.963, p value = 2.70E-20 (p < 0.001).

[0240] The exemplary embodiment of the system uses fixed prototype patterns, parameters and priors (i.e., class ω kThe prior probability (ω k )) for sensing and decision making, and the approach is a statistical risk-sensitive framework with a pre-adjusted risk matrix. A more comprehensive model of the system, in which the previous elements (especially the variables used to adjust the system) will be adaptable, can benefit from online learning and adaptation capabilities.

[0241] Among the areas of machine learning that can be applied to the current model, reinforcement learning can be implemented, where an agent that learns to perform specific actions that result in the highest reward is a subset of unsupervised learning that is causal. Q-learning, a form of reinforcement learning, has shown promise in modeling reward-based human decision making and dopaminergic-mediated reward prediction mechanisms.

[0242] For this implementation of Q-learning, it is useful to define a few concepts:

[0243] - Environment: the situation in which the problem occurred,

[0244] -Agent: an autonomous actor that performs actions under certain assumptions,

[0245] - Actions: any possible behavior that the agent can perform (like links in the diagram),

[0246] - Status: The situations that the agent can host (such as nodes in the figure),

[0247] - Reward: the immediate benefit granted by the environment to the agent for its last action (e.g., the weight assigned to each link),

[0248] - Policy: A strategy that dictates the agent's next action based on the current state,

[0249] - Value: the expected long-term benefit of using a particular policy in any state, and

[0250] - Action-Value: Also called Q-value, similar to value and also depends only on action, and means the expected long-term benefit of taking a specific action with a specific policy in a specific state.

[0251] If we assume that, in some environment, one of the states is a goal, then the goal of a clean agent would be to reach that goal state when placed in the environment in question. Designating a state as a goal is achieved by associating a reward value or weight with each possible action.

[0252] For example, if there is a state where the agent can reach the goal state in the next step, the reward value for that action is 100, but any action that does not lead to the goal state has a reward value of zero. You can think of the reward as a matrix that relates states to actions, where the elements of the matrix are the reward values. In this reward matrix, in addition to the values ​​zero and 100, when an action does not exist, we can only assign negative reward values ​​to it. The reward matrix is ​​initialized to a zero matrix at the beginning of learning.

[0253] Now, the action-value matrix Q with an unknown number of elements and also initialized as a zero matrix with a single zero value (the matrix will expand as new states are discovered during learning) represents the brain of our agent, who knows nothing at the beginning of learning.

[0254] A simple transfer formula is used:

[0255] Q(state,action)=Reward(state,action)+γ*Max(Q(next state,allactions))

[0256] Each element of the action-value matrix Q will be calculated by adding the learning term to the corresponding value in the reward matrix. The update term is the maximum value of all possible actions for the next state in Q multiplied by the learning rate γ.

[0257] The algorithm is started by initializing the agent in a random state and the action-value matrix Q to zero. At each step, the agent randomly chooses to reach one of the possible states, and then we calculate the value of the element in the action-value matrix Q by adding the corresponding element in the reward matrix to the maximum value of all possible actions in the next state in the action-value matrix Q, multiplied by the update factor. What has just been explained represents one iteration of the learning process, and the iteration is carried out in the direction of convergence. The corresponding learning process is as follows Figure 3 A simplified example of the learning steps for a three-state system with an absorbing target in is shown.

[0258] At convergence, it can be assumed that the agent knows the optimal path to the goal, meaning that, from any initial state, the path with the highest sum is the path to be taken, as proposed by McCullock, J. (2012) in "A Painless Q-Learning Tutorial", available at http: / / mnemstudio.org / path-finding-q-learning-tutorial.htm.

[0259] In our biological motion perception model framework, the agent will be the third layer of complex pattern detectors that are implemented in simulated complex pattern detection neuronal structures 6, 6L, 6R and recognize different phases of motion. The state space is the stimulus that covers various body movements and moves through the first and second layers, and the action space is the classification of observations into their corresponding time phases. At each run, we set up a new environment with a new initial state. At each step, for a given sequence in the stimulus, we get an action from the agent, based on which the environment returns a reward and a new state.

[0260] This set of states, actions, rewards, and next states forms a Q-value according to the mentioned update rule. A stable and converging Q-value is a policy that enables the agent to make the best classification given the observations to obtain the maximum reward. In order to obtain accurate Q-values, deep neural networks can be used. Hence the term Deep Q-learning.

[0261] To model systems with unknown transition probabilities like human behavior, a risk-sensitive Q-learning algorithm that implements sequential decision making can be used. The utility function is chosen to provide a risk-sensitive framework for decision making with noisy rewards, as proposed by Shen, Y (2015) "Risk-sensitive Markov Decision Processes" (doctor rerum naturalium), University of Berlin Institute of Technology.

[0262] The system can have multiple uses as a functional simulator. The system can be applied to optical ophthalmology prescription or design, because it allows inter-individual adaptation of optical devices on a simulation system. Based on the neural imitation model, by taking into account the individual adaptation ability, especially by taking into account τ a The values ​​and the corresponding layers that model the excitatory / inhibitory neuronal responses can improve the concepts and innovations of new optical designs or new coatings. Depending on the individual characteristics of the future wearer such as age, refractive error, expertise, etc., the adaptability of the system can be changed. By the system of the present invention that mimics the human brain, it is also possible to study the impact of ophthalmic lens distortion on the human decision-making process and model this impact.

[0263] As a result, ophthalmic lens manufacturers will be able to evaluate and estimate in advance the impact of their designs on real-life situations. Examples of this would be determining whether a person approaching from the front moves to the left or right of the observer to avoid a collision, or detecting more subtle movements. This application can be achieved by inserting a lens design model between the biological motion stimulus and the simulation model. By implementing this process before production, lens manufacturers can save a large part of the expense and time they would have invested in the evaluation phase.

[0264] For example, the effect of optical distortion on biological motion perception has been tested. The observed football shooting scene (i.e., the penalty kick display in the current example) has been distorted by 2.3 diopters of barrel aberration. The psychophysical functions were simulated and calculated in the system. This resulted in changes in the thresholds and slopes of the psychophysical functions of the simulated subject / system. For example, a simulated subject with a threshold of 9.3° and no distortion became 17.9° with distortion, and its slope changed from 0.12 to 0.06. For another simulated subject, the threshold changed from 10.7° to 42.3°, and the slope changed from 0.11 to 0.013. It is worth reminding that the better the accuracy of detecting the orientation of biological motion, the higher the slope, and vice versa.

[0265] In another application, the system can be applied to robot vision to quickly identify human motion gestures. For example, it can be applied to robots that help elderly people or other groups evaluate human-based motion. It can also be applied to monitoring systems to determine specific movements, or in sports for behavior monitoring.

[0266] The system according to the invention can be implemented in many other ways, for example according to the following possibilities:

[0267] In an exemplary embodiment of the system, the opponent motion detector, which simulates the opponent motion detection neuron structure, receives data only from the neighboring receptive fields of the local motion energy detector. In more advanced systems, data from two distant receptive fields can be pooled to be able to detect global or at least large relative motion in a moving scene.

[0268] In an exemplary embodiment of the system, both the first and second layers are simulated as noise-free, but as seen above, noise may be added to one or both of these layers.

[0269] Implementation methods for testing include: Figure 1 The system shown in FIG. 1 shows a system in which there is only one simulated visual path and interface and layer hierarchy in a monocular application. Figure 2As shown, the system can be extended to a binocular implementation, wherein two subsystems 1R, 1L, respectively composed of simulated visual pathways and interfaces 3R, 3L, simulated local motion direction detection neuron structures 4R, 4L, simulated opposing motion detection neuron structures 5R, 5L, and simulated complex pattern detection neuron structures 6R, 6L, are arranged in parallel, and the outputs of the two subsystems (the two simulated complex pattern detection neuron structures from the two subsystems 1R, 1L) are sent to a common simulated motion pattern detection neuron structure 7LR that provides decisions. Since the two simulated visual pathways and the two interfaces have two different perspectives toward the same scene, and since the parallel layers, additional decisions related to the motion characteristics available to binocular vision can be provided.

Claims

1. A system (1) for simulating a decision process in a mammalian brain regarding a motion characteristic associated with a body posture of a visually observed body by means of a simulated visual pathway including an interface to a simulated neuronal structure, the body being visually observed by the simulated visual pathway and its interface, the simulated visual pathway and its interface at least converting luminescence information of the observed body into an optical flow data stream, the optical flow data stream conveying visual observation information associated with the visually observed body, and the visual observation information being processable in the simulated neuronal structure, The system is a feed-forward system and the hierarchy from the visual observation to the decision-making includes: - the simulated visual pathway and its interface (3, 3L, 3R) configured to simulate an eye and a final ophthalmic device on said at least one eye and comprising a set of receptive fields, - simulate a local motion direction detection neuron structure (4, 4L, 4R) for detecting the motion direction through a receptive field and receiving an optical flow data stream from the interface, - a simulated opposing motion detection neuronal structure (5, 5L, 5R) for detecting opposing motions associated with at least expansion and contraction, and receiving an optical flow data stream from said simulated local motion direction detection neuronal structure, - an analog complex pattern detection neuronal structure (6, 6L, 6R) for detecting optical flow patterns globally over the entire visual observation and as a function of their evolution over a time period of said entire visual observation, and receiving an optical flow data stream from said analog opponent motion detection neuronal structure, the detectable patterns being prototype patterns, and - a simulated motion pattern detection neuron structure (7, 7LR) for detecting motion patterns and receiving an optical flow data stream from the simulated complex pattern detection neuron structure and providing a decision about motion characteristics, Wherein, the neurons of the simulated motion pattern detection neuron structure (7, 7LR) all include forgetting ability, and the forgetting ability is a function of delay and activity of the neurons.

2. The system according to claim 1, wherein: The simulated opposing motion detection neuronal structure (5, 5L, 5R) further detects the opposing motion associated with the rotation.

3. The system according to claim 1, wherein: Internal noise of the system is further simulated in the system, and wherein the noise is added to the optical flow data stream output from the simulated complex pattern detection neuron structure (6, 6L, 6R).

4. The system according to claim 1, wherein: The simulated local motion direction detection neuron structure (4, 4L, 4R) is configured to detect the motion direction in the two-dimensional space between the receptive fields, and the detectable motion directions are up, down, left, and right.

5. The system according to claim 2, wherein: The simulated oppositional motion detection neuron structure (5, 5L, 5R) is configured to detect expansion, contraction, clockwise rotation and counterclockwise rotation in two-dimensional space.

6. The system according to claim 1, wherein: The simulated complex pattern detection neuron structure (6, 6L, 6R) includes a set of simulated neurons and is configured to detect prototype patterns in a two-dimensional space within a global observation angle with a determined value, the simulated neurons of the simulated complex pattern detection neuron structure being assigned to a plurality of groups (10, 11) in the set, the number of the groups (10, 11) corresponding to the number of possible decisions that the system can provide, each group including simulated neurons that are asymmetrically and laterally connected together within the group.

7. The system according to claim 1, wherein: The simulated motion pattern detection neuron structure (7, 7LR) is configured as a network of mutually inhibitory simulated neurons (12, 13), wherein the simulated neurons with the highest excitatory inputs inhibit the activities of other simulated neurons in a nonlinear and interactive operation, wherein the activities of the other simulated neurons do not exceed their defined thresholds.

8. The system according to claim 7, wherein: The mutually inhibited simulated neurons (12, 13) of the simulated movement pattern detection neuron structure (7, 7LR) have additional inputs, through which the forgetting ability is implemented, and the additional inputs are directed to the neurons that are excited first according to Dis T = u ( t - τ a )∗[ T - S ( P T ( D ))- k Dis T ] to receive data, and for other neurons according to Dis D = u ( t - τ a )∗[ D - S ( P D ( D , T ))- k Dis D ] to receive data, and wherein T is the activity of the first neuron that is excited by the optical flow data stream received from the simulated complex pattern detection neuron structure, and D is the activity of the remaining neurons of the network, u() is a unit step function, τ a is the time constant, and S () is the modified Mie function or any other function having the same mathematical symmetry as the Mie function, and k Dis is the weighting coefficient.

9. The system according to claim 1, wherein: The system includes at least one programmable computer implementing the neuronal architecture.

10. The system according to claim 9, wherein: The at least one programmable computer comprises an input-output interface allowing input of parameter values ​​modifying at least a delay associated with the forgetting capacity.

11. The system according to claim 1, wherein: The system further includes a learning function implemented using a risk-sensitive Q-learning algorithm.

12. The system according to claim 11, wherein: The learning function is implemented in the simulated complex pattern detection neuron structure.