Task processing system and robot

Abstract

The application discloses a kind of bionic dynamic neural network, the neural network can form behavior state relation net by learning, and can be received when setting target task, state transfer and behavior activity control are carried out according to current state and the behavior state relation net formed, directly complete the target task.The bionic dynamic neural network provided in the embodiment of the application imitates the neural circuit topological structure and dynamic characteristics of biology, realizes the control to state transfer, so that state transfer and action control form the behavior state relation net with causal relationship, more in line with the way of human brain processing problem, improve the decomposition ability of complex task and the efficiency of reinforcement learning, so that reinforcement learning can be more realistically used in actual robot task.The bionic dynamic neural network of the embodiment of the application can be applied in reinforcement learning model, for building the reinforcement learning algorithm model with higher learning efficiency capable of decomposing task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent algorithm, in particular to a bionic dynamic neural network and a learning method and application thereof. BACKGROUND

[0002] Since deep reinforcement learning can enable robots to learn through trial and error, it provides strong algorithm support for precise intelligent control. In particular, with the rapid development and wide application of AI technology, the research on deep learning algorithms supporting the maturity of AI technology has increasingly become a new hotspot. However, the existing deep reinforcement learning algorithm has the following defects:

[0003] 1) The neural network model used by the existing deep reinforcement learning algorithm is a black box, and can only be learned end-to-end. This approach can cause the entire action to be discarded if a step is wrong, so the robot needs to perform a large number of attempts during training. The efficiency of the attempt and learning is low, and the convergence is slow, so the time required for the attempt and learning is usually much longer than the time the robot can run, and the efficiency is very low.

[0004] 2) Some popular deep reinforcement learning models, such as QNN, DDPG, A3C, TRPO, etc., are all based on artificial neural networks and are trained in a virtual simulation environment. However, since the virtual simulation environment cannot fully cover the possible situations in the real environment, the application effect of such models is not satisfactory when transplanted to the real environment. SUMMARY

[0005] In order to solve the above problems, the inventors have found the root cause of the above defects in the current mainstream deep reinforcement learning algorithm through a large number of research and experimental attempts, which is that the model of the neuron and the connection between the neurons is a static function mapping rather than a dynamic system, resulting in low efficiency when processing continuous input signals; its learning process completely depends on external rewards, resulting in learning only when the external reward signal is provided; its topological structure focuses on end-to-end mapping while ignoring the mutual influence between neurons in the same layer, so it is difficult to form the causal relationship between actions and states, and thus cannot optimize complex tasks.

[0006] Based on this, the inventors conceived to construct a new type of bionic dynamic neural network by simulating the brain of a living being, to realize efficient processing and analysis of continuous signals by means of a compatible dynamic model of neurons. The inventors also conceived to design a mechanism for intrinsic reward and curiosity-driven learning in this new type of neural network, to encode the state of the robot and the environment and to calculate curiosity by means of an interpretable neural network topology, to realize the prediction of consequences from a given action and the selection of an action from a given consequence by means of the causal relationship between state and action, to realize the decomposition of complex tasks and the sequencing of actions in a state path search and action regulation corresponding to the state, and thus to improve the learning efficiency.

[0007] According to a first aspect of the present application, a bionic dynamic neural network is provided, wherein the bionic dynamic neural network is capable of forming a behavior-state relationship network by learning, and is capable of performing state transition and behavior activity regulation according to a current state and the formed behavior-state relationship network upon receiving a set target task, to directly complete the set target task.

[0008] According to a second aspect of the present application, another bionic dynamic neural network is provided, which comprises at least one network unit having a plurality of shared state neurons, each network unit comprising a behavior neuron,

[0009] wherein the state neurons and the behavior neurons are capable of forming a behavior-state relationship network by learning;

[0010] The state neurons regulate the activity of the behavior neurons and realize state transition through the behavior-state relationship network.

[0011] According to a third aspect of the present application, yet another bionic dynamic neural network is provided, which comprises at least one network unit having a plurality of shared state neurons, each network unit comprising a behavior neuron and a reward neuron,

[0012] wherein the state neurons and the behavior neurons are capable of forming a behavior-state relationship network by learning;

[0013] The reward neuron regulates the learning process of the network unit in which it is located according to an internally generated excitation signal or an externally received excitation signal;

[0014] The learning mode comprises learning based on spontaneous exploration of the behavior neuron and internally generated excitation signals, and learning based on received external excitation signals.

[0015] According to a fourth aspect of the present application, a learning method of the above-mentioned bionic dynamic neural network is provided, which comprises:

[0016] The biomimetic dynamic neural network is initialized so that a connection relationship is established between all state neurons and all reward neurons;

[0017] The input signal is received to perform low-dimensional signal to high-dimensional space mapping through the mapping neuron, and the corresponding state neuron is activated, and the mapping rule is randomly generated according to a certain principle in the initialization process, for example, in a manner of randomly sampling signal combination;

[0018] When the external reward signal does not exist, exploration learning is performed based on spontaneous excitation of the action neuron, the behavior neuron is randomly activated, for example, the behavior neuron generates spontaneous excitation or is excited due to random reception of the signal of the state neuron, and the reward neuron releases the modulatory substance according to the excitation of the state neuron in the network unit in which the behavior neuron is in the excited state, for example, the state neuron can activate the reward neuron to release the modulatory substance when the state neuron is in the initial excitation state for several times or is in the excited state and has not been excited for a long time;

[0019] When the external reward signal exists, the reward neuron releases the modulatory substance according to the received external reward signal;

[0020] The modulatory substance is used to strengthen the lateral connection of the last activated state neuron and the currently activated state neuron and the secondary connection between the lateral connection and the behavior neuron in the network unit, and is used to weaken the connection between the currently activated state neuron and the reward neuron.

[0021] According to a fifth aspect of the present application, an electronic device is provided, which comprises at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method or realize the functions of the above biomimetic dynamic neural network.

[0022] According to a sixth aspect of the present application, a storage medium is provided, which stores a computer program executable by a processor to realize the steps of the above method or realize the functions of the above biomimetic dynamic neural network.

[0023] The bionic dynamic neural network provided by the embodiment of the present application imitates the neural circuit topology structure and dynamic characteristics of a biological organism, realizes control on state transition, forms a behavior state relationship network with a causal relationship between state transition and action regulation, is more in line with the way in which the human brain processes problems, improves the decomposition ability for complex tasks and the efficiency of reinforcement learning, and enables reinforcement learning to be more realistically used in actual robot tasks. The bionic dynamic neural network of the embodiment of the present application can be applied to a reinforcement learning model, and is used to construct a reinforcement learning algorithm model with higher learning efficiency for decomposing tasks. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A topology structure diagram of the bionic dynamic neural network of an embodiment of the present application;

[0025] Figure 2 A topology structure diagram of the bionic dynamic neural network of another embodiment of the present application;

[0026] Figure 3 A topology structure diagram of the bionic dynamic neural network of still another embodiment of the present application;

[0027] Figure 4 A structure schematic diagram of each network unit of the bionic dynamic neural network of an embodiment of the present application;

[0028] Figure 5 A flowchart of a learning method of the bionic dynamic neural network of an embodiment of the present application;

[0029] Figure 6 A system framework diagram in which the neural network of the embodiment of the present application is applied to a task environment is schematically shown;

[0030] Figures 7 to 10 A learning task is to push a ball into a hole by holding the ball and avoiding obstacles by a mechanical arm, and a difference between the exploration learning process of the neural network of the embodiment of the present application and a typical deep reinforcement learning exploration process is schematically shown Figures 8 to 10 Figure 7

[0031] Figure 11 A structure schematic diagram of an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0032] The present application will be further described in detail below with reference to the accompanying drawings. Figure 1 A topology structure of the bionic dynamic neural network of an embodiment of the present application is schematically shown, as Figure 1 ​​As shown, the biomimetic dynamic neural network 1 includes a plurality of network units 11 (the number of network units can be set according to requirements in other embodiments) and a mapping neuron 13 for signal mapping, each network unit 11 includes a behavior neuron 11A for responsible action output, and a reward neuron 11B for regulating the learning process inside the network unit. All network units 11 share a plurality of state neurons 14.

[0033] In the embodiment of the present application, the mapping neuron maps the low-dimensional signal to the high-dimensional state space by sparse coding, and realizes the encoding and storage of the low-dimensional signal in the state neuron group in the form of sparse coding. The mapping of the embodiment of the present application realizes the mapping of the low-dimensional signal to the state neuron by random mapping input, in particular, by random mapping in the form of randomly extracting the combined signal, and the random mapping is realized by the form of sampling combination addition, so that a small part of the state neurons are activated. The information of the random mapping input is the potential of the mapping neuron and its change over time. It should be noted that the sparse coding here is a general sparse coding, and the sparse coding can realize the mapping of the low-dimensional coded information to the high-dimensional space, encode the information by a set of "super-complete basis vectors", and reduce the overlap degree of different information. By the mapping mode of sparse coding, the information is preserved in the state neuron in the form of sparse coding, and the overlap degree between them is low, so when the connection strength between a certain state neuron and a certain behavior neuron changes, only the relationship between this state and this behavior changes, and other states and this behavior will not be disturbed, thus reducing the interference of learning different "knowledge points", greatly reducing the risk of catastrophic forgetting, and improving the learning efficiency.

[0034] The state neurons and the behavior neurons in the embodiments of the present application can form a behavior state relationship network through learning. The behavior state relationship network is used to identify the causal relationship between the states and the behaviors, and can be used to regulate the activities of the behavior neurons to realize action switching and the state transition caused thereby. The specific learning process of the neural network will be described in detail below. By way of example, the behavior state relationship network formed through learning includes lateral connections formed between the state neurons to describe the state transition path and secondary connections formed between the lateral connections and the behavior neurons; or includes lateral connections formed between the state neurons to describe the state transition path, secondary connections formed between the lateral connections and the behavior neurons, and connections formed between the state neurons and the behavior neurons. The lateral connections in the computational neuroscience refer to the connections between the same neurons. The state neurons form a layered structure in each network unit through the lateral connections, and thereby form an RNN (Recurrent Neural Network) structure in the entire neural network. In particular, the lateral connections in the embodiments of the present application are used to describe the state transition path. The state neurons located downstream of the lateral connections can be activated by the state neurons located upstream to transfer the reflected state, thereby the state neurons can influence each other. By establishing the lateral connections between the state neurons respectively, these state neurons and their connection relationship can form a structure similar to a graph, and realize the complex state transition effect beyond the Markov model (a statistical model widely used in the field of natural language processing). For example, in the Markov model, only one state neuron is in an activated state, while in the neural network structure of the embodiments of the present application, through the sparse coding manner to retain information in the state neurons and the mutual influence between the state neurons, multiple state neurons can be activated at the same time.

[0035] In the embodiment of the present application, each network unit of the neural network further comprises a behavioral neuron, which is used to adjust downstream neural circuits. Specifically, the behavioral neuron can control the output action (such as the action of a robot) by controlling the lower-level neural network, and change the state (such as the state of the robot or the environment) through the output action. These states can be sensed by sensors, and then mapped to the state neurons through the mapping neurons and activate the corresponding state neurons, thereby updating the current state. In addition, the behavioral neuron in the embodiment of the present application also generates a secondary connection with the lateral connection between the state neurons. In this way, the state neurons can activate the behavioral neurons through the direct connection or the secondary connection with the behavioral neurons. Since the neural network in the embodiment of the present application comprises a plurality of network units, each network unit comprises a different behavioral neuron, and the network units share the state neurons, the state neurons can activate different behavioral neurons, thereby generating diverse behaviors. It should be noted that the activation and excitation in the embodiment of the present application are both computational neuroscience terms. The activation of a neuron means that the excitation of other neurons leads to the excitation of the neuron, and the excitation of the neuron means the increase of the neuron potential or the generation of a pulse. The behavioral neuron in the embodiment of the present application usually needs to be excited together with a plurality of state neurons (of course, it can also be excited by one state neuron in some embodiments), and the output of the behavioral neuron when excited is related to the input of the state neurons to the behavioral neuron and the parameters of the behavioral neuron itself.

[0036] In the embodiment of the present application, each network unit of the neural network further comprises a reward neuron for regulating the learning process. The regulation of the learning process by the reward neuron comprises releasing a modulatory substance inside the network unit according to the reward signal and the activity of the state neurons, so as to dynamically adjust the connection weights between the state neurons and the behavioral neurons. The activity of the state neurons refers to the change of the potential, which is generally represented by a set of differential equations in computational neuroscience. In specific implementation, the adjustment of the connection weights between the state neurons and the behavioral neurons is jointly affected by the modulatory substance released by the reward neuron and the activity of the connected neurons, and is described by a synaptic plasticity model. The adjustment of the weights can be implemented in various computational neuroscience synaptic plasticity models, such as a dynamic synapse model, a modulated STDP model, and a hedonic synapse model.

[0037] As a preferred implementation example, the reward signals in the neural network of the embodiment of the present application include two kinds, one is the reward signal input from the outside world, and the other is the reward signal generated internally. Among them, the reward signal input from the outside world will directly cause the reward neuron to be excited and release the modulatory substance; while the generation of the internal reward signal can be the activation from the new state and the reduction of the distance between the current state and the target state, which is calculated by the reward neuron according to the potential signal of the state neuron. The reward neuron can control the learning process through the released modulatory substance, which is similar to the “dopamine” regulating the learning process of animals, or directly regulate the learning process through the external reward signal. Specifically, in the embodiment of the present application, the regulation process can be realized as controllably changing the parameters in the differential equation of the neuron. In this way, the reward neuron can regulate the learning process in the network unit it is in according to the internal generated excitation signal or the excitation signal received from the outside.

[0038] In the neural network of the embodiment of the present application, through learning, there is a lateral connection between the state neurons in each network unit, and the secondary connection is formed between the behavior neuron and the lateral connection. According to the model characteristics and the learning situation, each state neuron can be connected with the behavior neuron, but whether the connection is connected and the strength of the connection is related to learning. Therefore, the behavior-state relationship network can be formed in the neural network through learning, and when the task comes, the action output and state transition can be performed based on the formed behavior-state relationship network, so as to achieve the effect of completing the target task based on the decomposed action and state, which is beneficial to complete the complex task through coordinating multiple simple networks, and is more consistent with the biological brain circuit, and has high learning efficiency. In addition, in the embodiment of the present application, there is no behavior neuron and reward neuron from other units in each network unit, and there is only one behavior neuron in a unit by default, and there are multiple state neurons in different units, so that one behavior can be awakened through multiple states, and different behaviors can be generated through the same state, thereby corresponding to multiple behaviors, and the mutual influence of state and behavior can be realized, so that the state and behavior in the generated behavior-state relationship network have a causal relationship, and the efficiency and reliability of the neural network model are improved.

[0039] Figure 2 and 4 The topological structure of the bionic dynamic neural network of another embodiment of the present application is schematically shown, Figure 4 The structure diagram of each network unit formed by learning is shown in the figure, in which, Figure 2 and 4In another preferred implementation, the neural network of the embodiment of the present application further comprises a target neuron 12 for setting a target state, wherein the target neuron 12 can be set in each network unit, or all network units share one target neuron, and in the embodiment of the present application, it is preferred to share one target neuron, which can establish a connection with a specific state neuron through learning. As shown in the left part of Fig. 1, Figure 4 The left part of Fig. 1 shows the structure of each network unit. As shown in the left part of Fig. 1, in different units, the state neurons 14 are the same, the behavior neurons 11A and the reward neurons 11B are different, and the target neurons 12 are usually different. There is a connection between the behavior neurons 11A and the state neurons 14, as shown by the orange dashed connection line in the figure. There is a lateral connection between the state neurons, as shown by the black arrowed connection line in the figure, which can be used to build a model of state transition. There is a secondary connection between the lateral connection and the behavior neuron, as shown by the red solid connection line in the figure, which can be used to adjust the behavior activity. Among them, there is a connection between the target neuron and a specific state neuron, as shown by the green connection line with a dotted head in the figure, through which the specific state neuron can be set as a target state. The reward neuron obtains the progress information of the target completion from the state neuron, releases the reward, and regulates the learning of this unit and the corresponding lower neural network of this unit. In different units, the neuron connection is subject to the same rules, but the specific connection is different. Figure 4 The right part of Fig. 1 shows the lateral connections existing in two network units, represented by black solid lines and yellow dashed lines respectively. The two groups of lateral connections represent the state transitions caused by two behaviors.

[0040] In the preferred implementation, the state neurons are configured to include signal potentials for encoding signaling and task potentials for defining state targets. The target neuron is configured to achieve the setting of state targets by controlling the task potentials of certain state neurons, such as lowering the task potentials of certain state neurons. Here, the certain state neurons refer to the state neurons as the target state. The neural network of the embodiment of the present application can learn to establish connections between the target neuron and the certain state neurons, so that upon receiving a target task, the target can be set by the target neuron changing the task potentials of the certain state neurons. Taking the target setting by raising the task potentials as an example, since the lateral connections are formed between the state neurons through learning, as the task potentials of the certain state neurons are raised, a plurality of groups of task currents across the state neurons along the lateral connections are generated between the certain state neurons and other state neurons connected laterally, and the plurality of groups of task currents generated can indicate a plurality of paths from the current state to the target state (the task currents indicate the possible state paths from the current state to the target state according to the connections between the state neurons), and the strength of the task currents can indicate the strength of the intention to make state transitions along the paths. Based on this, after the target neuron sets a state target for a certain input task according to the learning results, the neural network of the embodiment of the present application can use the task potentials and the behavior-state relationship network formed to search for paths from the current state to the target state, and according to the path search results, a plurality of groups of task currents for indicating the possible state paths from the current state to the target state are established. Through the transitions of the states on the paths and the triggering of the behaviors, the decomposition of the input task can be achieved. Here, the across refers to the current loop caused by the task potentials passing through a plurality of state neurons.

[0041] In a preferred embodiment, lateral inhibition can also be set between the behavioral neurons. In this way, when there are more than two network units, the behavioral neurons in different network units can select behaviors based on the lateral inhibition due to the lateral inhibition between them. Thus, the neural network can also use the task current across the state neurons formed in the behavioral state relationship network by the task potential of the state neurons and the lateral inhibition between the behavioral neurons to select behaviors. Exemplarily, the behavioral neurons are mathematically implemented by a differential equation, and since the secondary connections are formed between the behavioral neurons and the lateral connections in the behavioral state relationship network, in a specific implementation, the current intensity that the behavioral neurons can contact through the secondary connections can be taken as the input of the differential equation to control the excitement degree of the behavioral neurons, and the lateral inhibition of the behavioral neurons can be implemented as inhibition between the behavioral neurons when the excitement potential is generated, for example, the potentials generated by the behavioral neurons are weighted and subtracted from each other repeatedly, and the behavioral neurons whose final subtraction result is greater than a certain threshold can output actions. Due to the lateral inhibition between the behavioral neurons, usually only the neuron with the strongest activity can output.

[0042] Since the state neurons represent some states and the lateral connections represent the transition paths between the states, based on these states and state transition paths, the task potential can be used to search for a feasible transition path from the current state to the target state. And because each state transition corresponds to a subdivision of a task, a corresponding behavior or action is needed to complete, the behavioral neurons required for the current task subdivision can be selected and activated through the task potential. In this way, after the state target is set, the state routing and action selection can be completed through the task potential.

[0043] In another preferred embodiment, the learning of the neural network can be implemented in two modes, exemplarily, one can be a curiosity mode and the other can be an external excitation mode. Among them, the curiosity mode refers to learning based on the spontaneous exploration and generated internal excitation signal of the behavioral neurons; the external excitation mode refers to learning based on the received external excitation signal. In order to realize the learning of the curiosity mode, the behavioral neurons can be configured to be able to realize spontaneous excitement, so that the exploratory learning based on curiosity can be caused by the spontaneous excitement of the behavioral neurons. Specifically, the spontaneous excitement of the behavioral neurons can be realized by setting the behavioral neurons as a model capable of generating random excitement, or setting the behavioral neurons as a model capable of generating unstable connections with the state neurons.

[0044] As another preferred embodiment, in the bionic dynamic neural network, a connection can be generated between the state neuron in the excited state and the reward neuron upon receiving an excitation signal from the outside. Thus, the effect of avoiding the instability of external reward can be achieved. Preferably, taking the example of setting a goal by pulling up the task potential of the target neuron, since a plurality of task currents spanning the state neurons via the lateral connections are generated between the specific state neuron and other state neurons with lateral connections through path searching, and the potential is applied to the network similar to a resistor, i.e., the state neurons, the potential of the state neuron corresponding to the current state is higher on the path of these task currents as the current state approaches the target state, i.e., when the currently excited state neuron is closer to the target state than the last excited state neuron, the task potential of the currently excited state neuron is higher than that of the last excited state neuron. Thus, the reward neuron can calculate the distance between the current state and the target state by monitoring the change in the difference between the task potential of the state neuron of the current state and the task potential of the state neuron of the target state, and calculate the reward and release modulator according to the change in the difference, for example, release the reward when the difference decreases, and do not release the reward when the difference does not change or increases, to achieve sub-task reward (release the reward when the state is closer to the target task, i.e., sub-task reward).

[0045] In a preferred implementation, the neural network is further configured to further optimize the state path when a shortcut is found. Since the reward neuron can calculate the reward by monitoring the change in the distance between the current state and the target state, when a shortcut is found (i.e., when the current state neuron is found to make the current state closer to the target state faster), the reward neuron will release a stronger reward, so that the strength of the lateral connection corresponding to the shortcut increases, making it more likely to be used in future state routing. Conversely, the use of long paths will gradually decrease.

[0046] Through the activation control of the state neurons in the above-mentioned embodiments of the application, since the information is preserved in the state neurons in a sparse coding manner and the lateral connections can be formed between the state neurons, the activated state is transferred between different state neurons as time or input signals change, and only a small number of state neurons are in the activated state at the same time. A small number means that the number of state neurons activated at the same time is not less than one and not more than half of the total number of state neurons in each layer structure, preferably much less than half of the total number of state neurons in each layer structure, such as one-twentieth of the total number of state neurons.

[0047] Figure 3 The topological result of the bionic dynamic neural network of another embodiment of the application is schematically shown as Figure 3As shown, in the neural network, a preprocessing unit 10 for transforming the input signal to output a low-dimensional signal to the mapping neuron is further included. In this way, after the signal received by the sensor enters the neural network of the embodiment of the present application, it is first preprocessed (e.g., feature extraction) by the preprocessing unit according to the specific task and the form of the input signal, so that the input signal is converted into a signal that is more easily used by the neural network, which can improve the applicable scenarios of the neural network of the embodiment of the present application and reduce the requirements for the input signal. Exemplarily, the sensor can be a visual sensor, a joint angle sensor, a distance sensor, a force sensor, a Lidar, etc. Exemplarily, the signal (such as a sensor signal) input to the neural network is subjected to nonlinear transformation by the nonlinear transformation of the preprocessor (e.g., a computational neurodynamic model, a deep network model, or other non-neural network model for nonlinear transformation processing) to obtain a low-dimensional signal output to the mapping neuron, which is then mapped to a high-dimensional space by the mapping neuron.

[0048] Preferably, the state neuron of the embodiment of the present application extends a large number of parallel axons through a plurality of similar network units, so that the internal state, input and output of the state neuron are shared by the plurality of network units, so that all network units can obtain information of these states.

[0049] It should be noted that all the neurons mentioned in the above embodiments of the present application can be implemented by a neuron model with dynamic characteristics in computational neurology, such as Izhikevich Neuron (Izhikevich neuron model), Hodgkin–Huxley model (Hodgkin-Huxley model), FitzHugh–Nagumo model (FitzHugh-Nagumo model), etc. This means that the state of the neuron can be unstable, but can dynamically change state. Among them, the state transition can be influenced by the state neurons through lateral connections, or can be regulated by the behavior neurons based on the selected downstream neural circuit.

[0050] The neural network of the embodiment of the present application can form a dynamic routing network based on the mutual influence of the state neurons, to complete the switching of the state according to the execution of the action and the change of the external input, and to realize the decomposition of the complex task. Moreover, the target state can be activated by setting the target neuron, and the state can be close to the target state through the action exploration of the behavior neuron, and when the target state is approached, the reward neuron is excited by the target neuron to release the reward, so that the connection between the neurons is updated to remember the relevant state path, thereby learning the causal relationship between the action and the state, improving the learning efficiency, and making the reinforcement learning more easily performed on the physical robot.

[0051] Figure 5 The learning method of the biomimetic dynamic neural network of the above embodiment of the present application is schematically shown as follows, Figure 5 The method comprises the following steps:

[0052] Step S501: initialize the above biomimetic dynamic neural network, so that all the state neurons and all the reward neurons are connected.

[0053] Step S502: receive the input signal to map the low-dimensional signal to the high-dimensional space through the mapping neuron, and activate the corresponding state neuron through the random mapping input.

[0054] Step S503: when the external excitation signal is absent, learn through the curiosity learning mode. Specifically, the state neuron and the behavior neuron in the excited state can be activated through the unstable connection or the spontaneous excitation of the behavior neuron, and when the behavior neuron is in the excited state, the modulatory substance is released according to the excitation of the state neuron in the network unit where the behavior neuron is located, so as to strengthen the lateral connection between the last activated state neuron and the current activated state neuron and the secondary connection between the behavior neuron pointed by the lateral connection, and weaken the connection between the current activated state neuron and the reward neuron. In the initial state, all the state neurons are connected with the reward neurons, and when the curiosity mode is learned, the connection between the state neurons and the reward neurons is always weakened, and the connection between the state neurons and the behavior neurons, the lateral connection between the state neurons, and the secondary connection between the lateral connection and the behavior neuron are adjusted according to the modulatory substance and the selected synaptic plasticity model to adjust the strength of the connection, so that the correct connection between the action and the state transition is enhanced, and the incorrect connection is weakened, thereby forming a behavior-state relationship network with causal relationship. Correct means that the current state is more likely to approach the target state (which can be judged by calculating the difference between the task potential of the current state and the task potential of the target state), and incorrect means that the distance between the current state and the target state is more likely to increase.

[0055] Step S504: when the external reward signal exists, learn through the external reward mode. Specifically, the reward neuron releases the modulatory substance according to the received external reward signal, so as to strengthen the lateral connection between the last activated state neuron and the current activated state neuron and the secondary connection between the behavior neuron pointed by the lateral connection, and weaken the connection between the current activated state neuron and the reward neuron.

[0056] For example, the strengthening mode is to increase the connection weight, and the weakening mode is to decrease the connection weight.

[0057] More preferably, in step S504, the neural network can also strengthen or form the connection between the activated state neuron and the reward neuron according to the received external reward signal when the external reward signal exists.

[0058] It should be noted that the release of the modulatory substance of the reward neuron will always weaken the connection between the activated state neuron and the reward neuron, regardless of whether the reward neuron is excited due to internal curiosity or external reward, but the external reward will more strongly strengthen or form the connection between the activated state neuron and the reward neuron.

[0059] In the embodiment in which the bionic dynamic neural network includes the target neuron, the learning method of the bionic dynamic neural network further includes:

[0060] In the above step S501, when initializing the above bionic dynamic neural network, the corresponding state can be set as the target by establishing a connection relationship between a certain state neuron and the target neuron. In this way, when it is known which state neuron represents the target state, the learning and behavior of the device such as the robot carrying the neural network can be more directly controlled by directly connecting the target neuron. Of course, in specific implementation, in order to avoid more complex situations, the processing can also be arranged after the curiosity learning, i.e., step S503, and the embodiments of the present application do not limit this.

[0061] After the above step S504, it further includes that the reward neuron adjusts the connection weight between the target neuron and the reward neuron and the current state neuron according to the existence of the external reward signal, and when the external reward signal exists, the connection weight is enhanced, and when the received external reward signal in the same state continuously disappears, the connection weight is gradually weakened until the connection disappears.

[0062] In other implementations, in the embodiment in which the bionic dynamic neural network includes the target neuron, the learning method of the bionic dynamic neural network can also be that, in the initialization, the connection between the target neuron and the state neuron is not established, and only in step S504, the connection between the target neuron and the current state neuron is adjusted according to the external reward signal.

[0063] Through the above learning process, the behavior-state relationship network can be formed in the above dynamic neural network of the embodiments of the present application, so as to realize the decomposition of the set task in the manner of action regulation and state transition based on action, until the set task is completed, and the learning efficiency is improved. Moreover, the states and actions in the behavior-state relationship network formed thereby form a causal relationship, which is more in line with the operation mode of the brain circuit, and the execution efficiency and correctness of the task are higher.

[0064] Figure 6 The diagram schematically illustrates a system framework for applying the neural network of an embodiment of the present invention in a task environment, such as... Figure 6 As shown, by connecting the neural network with the downstream neural network (i.e., the secondary network) used to control the output action through behavioral neurons and reward neurons, the secondary network can be controlled to perform action regulation by regulating the output signal of the behavioral neurons. Based on the action regulation, the robotic arm or robot used to perform the task will produce corresponding actions. The execution of the actions will bring about changes in the state of the robotic arm and its environment. These changes affect the state changes through the neural network, which in turn affect the execution of the actions until the task is completed.

[0065] Figures 7 to 10 The illustration schematically demonstrates the exploratory learning process of the neural network in this embodiment of the invention and its differences from typical deep reinforcement learning in an application scenario where the learning task is to push a ball through a black obstacle held by a robotic arm into a hole. Figure 7 As shown, in typical deep reinforcement learning, the robot starts from a default starting point and attempts again. If it fails (as indicated by the red dotted arrow in the figure), the robot's position is reset, and it continues to try; no reward is given in this case. If it succeeds (as indicated by the green dotted arrow in the figure), it receives a reward and learns the attempted action. After accumulating enough successful attempts, it learns the correct action (as indicated by the purple continuous line arrow in the figure) and stores it. However, in the neural network model of this embodiment of the invention, as... Figures 8 to 10 As shown, its exploration process is based on a combination of curiosity-based learning and external reward mechanisms. That is, in the initial stages without task rewards, the robot learns... Figure 8 As shown, one can explore based on curiosity and learn the transition relationships between different states (as indicated by the dashed blue arrows formed by short line segments in the figure), gradually building a network of behavioral state relationships; when external rewards are obtained, such as Figure 9 As shown, the correlation between memory reward and state is illustrated, and alternative paths from the current state to the target state are searched from the state transition relationships (as indicated by the dotted purple arrows formed by the broken points in the figure), forming a behavioral state relationship network; and through continuous learning and discovery of shortcuts, such as Figure 10 As shown, the behavioral state relationship network can be further optimized to learn the optimal path from the initial state to the target state (as indicated by the continuous blue arrows in the figure). It is evident that existing reinforcement learning algorithms do not segment the data and lack internal rewards; they can only learn effectively when the target is successfully achieved. Figure 7 The neural network in this embodiment of the invention can divide the task into multiple segments, explore and learn segment by segment, and avoid repeated attempts or wasting already attempted actions. Figure 8 The specific learning process is combined with Figure 5 and Figures 8 to 10The method process shown can be:

[0066] In step S502, after the neural network model is initialized, the mapping neurons and the state neurons are connected according to a specific random rule, so as to randomly activate the state neurons according to the input signals. Since the neural network model has a certain ability to distinguish different states through sparse coding, the current state can include the posture of the robot, whether the ball is held up, the position of the ball, the joint speed of the robot, and the like according to the sensor signals of the robot.

[0067] In step S503, when it is judged that the external reward does not exist (judged by whether the reward neuron receives a certain reward signal of the external input, if received, it is considered to exist, if not received, it is considered to not exist, and in the model initialization stage, it is defaulted to not exist, the reward neuron in the embodiment of the application can calculate the reward according to the distance reduction between the current state and the target state, or can pass the external reward), the neural network model learns in a curiosity mode. Specifically, through the spontaneous random excitation of the behavior neuron or the unstable connection of the state neuron to the behavior neuron, the robot will make a new action, causing the change of the state of the robot or the operating object. Through the sensor, the changes of the robot and the operating object are perceived, and through the preprocessor and the mapping neuron, the new state neuron is activated. In this mode (i.e., in the mode of curiosity learning), in the network unit in which the behavior neuron is excited, if the excited state neuron is in the first several excitations or has not been excited for a long time, the state neuron will activate the reward neuron to release the modulatory substance. The modulatory substance will strengthen the lateral connection between the last excited state neuron and the current excited neuron and the secondary connection between the behavior neuron pointed by the lateral connection, which is the learning due to curiosity. Correspondingly, the modulatory substance will also weaken the connection between the state neuron and the reward neuron, which is the disappearance of curiosity for a specific state. Through the curiosity learning mode, the robot will remember the transition between states and the action that causes the transition of the state. The curiosity disappearance mechanism can avoid repeated learning due to curiosity. For example, the robot will learn how to make a new posture by trying to rotate the joint to a new angle. Through learning, the connections between the state neurons and the behavior neurons are regulated and optimized, and have the characteristics of being applied in a specific scene. In this way, the behavior neuron can control the robot through the control of the lower-level neural network. The action of the robot will further affect the state of the robot and its environment, and these states affect the excitation of the corresponding state neurons through the sensor, the preprocessor and the mapping neuron, and regulate the activity of the behavior neuron, so as to realize the tracking and transition of the state and the attempt of different actions.

[0068] In step S504, when judging that the external reward exists, the reward neuron is activated and in an excited state according to the received external excitation signal, and releases a modulatory substance to strengthen the lateral connection between the last excited state neuron and the current excited neuron and the secondary connection between the behavior neuron pointed by the lateral connection, and to weaken the connection between the current state neuron and the reward neuron. At the same time, in this mode, according to the existence of the external excitation signal, the target neuron also forms a connection with the current state neuron to set a task target, and the reward neuron also enhances or forms a connection with the current state neuron to remember which state corresponds to the external reward, so as to reduce the potential instability of the external reward. If the external reward continues to disappear in the same state, that is, the state with the external reward no longer receives the external reward, the connection between the target neuron and the state neuron and the connection between the state neuron and the reward neuron will gradually weaken until the connection disappears. This mechanism helps the robot to maintain the state of learning when the reward is unstable. After completing the basic learning in the above manner, after receiving the external reward once, the target neuron can set the target state through the learned bionic dynamic neural network to establish a task current and obtain a possible state path from the current state to the target state. The possible state path can provide a task segmentation scheme and a sub-task reward through each state on the path, and can also provide a basis for further optimizing the state path. Among them, each state on the path is the segmentation of the task; the reward is released when the state of the robot is closer to the target task, that is, the sub-task reward; and the state path can be further optimized when a shortcut is found.

[0069] It should be noted that different state transitions require different behavior neurons to be activated. By activating the behavior neuron, the downstream neural circuit or deep learning model can be further activated. The modulatory substance released by the reward neuron also affects the optimization of the downstream model. The specific model and optimization method can be selected according to the task. This mechanism can enable complex tasks that require a single complex network to be solved using multiple simple networks, reduce useless attempts through causal inference of actions and states, and improve network reuse rate and learning efficiency.

[0070] The above bionic dynamic neural network provided by the embodiments of the present application can be applied to a reinforcement learning algorithm model to realize intelligent control of mechanical or robotic actions through reinforcement learning.

[0071] In some embodiments, the embodiments of the present application provide a non-volatile computer readable storage medium, which stores one or more programs including execution instructions in the storage medium, the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, a network device, or an embedded system, etc.) to execute the learning method of the above bionic dynamic neural network of the present application.

[0072] In some embodiments, the present embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the learning method of the above-mentioned biomimetic dynamic neural network or implement the function of the above-mentioned biomimetic dynamic neural network.

[0073] In some embodiments, the present embodiments also provide an electronic device comprising at least one processor and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the learning method of the above-mentioned biomimetic dynamic neural network or implement the function of the above-mentioned biomimetic dynamic neural network.

[0074] In some embodiments, the present embodiments also provide a storage medium having a computer program stored thereon, which, when executed by a processor, can perform the learning method of the above-mentioned biomimetic dynamic neural network.

[0075] The biomimetic dynamic neural network of the above-mentioned embodiments of the present application can be used to perform the learning method of the biomimetic dynamic neural network of the present embodiments, and accordingly achieve the technical effects achieved by the learning method of the biomimetic dynamic neural network of the above-mentioned embodiments of the present application, which will not be repeated here. In the present embodiments, the relevant functional modules can be realized by a hardware processor.

[0076] Figure 11 is a hardware structure schematic diagram of an electronic device for performing the learning method of the biomimetic dynamic neural network according to another embodiment of the present application, as shown in Figure 11 The device comprises:

[0077] one or more processors 510 and a memory 520, Figure 11 In the above-mentioned embodiments, the processor 510 is taken as an example.

[0078] The device for performing the learning method of the biomimetic dynamic neural network can further comprise an input device 530 and an output device 540.

[0079] The processor 510, the memory 520, the input device 530 and the output device 540 can be connected through a bus or other means, Figure 11 In the above-mentioned embodiments, the connection through the bus is taken as an example.

[0080] The memory 520, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the voice recognition conversion method in the embodiments of the present application. The processor 510 executes various functions of the server and data processing by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, implements the learning method of the bionic dynamic neural network in the above method embodiments.

[0081] The memory 520 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the bionic dynamic neural network, etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 520 can optionally include a memory remotely arranged with respect to the processor 510, and these remote memories can be connected to the device carrying the bionic dynamic neural network through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0082] The input device 530 can receive input digital or character information, and generate signals related to user settings and function control of the bionic dynamic neural network. The output device 540 can include a display device such as a display screen.

[0083] The one or more modules described above are stored in the memory 520, and when executed by the one or more processors 510, the learning method of the bionic dynamic neural network in any of the above method embodiments is executed.

[0084] The above product can execute the method provided by the embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0085] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0086] (1) Mobile communication device: the feature of this kind of device is to have mobile communication function, and to provide voice and data communication as the main target. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0087] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. Such terminals include PDA, MID and UMPC devices, such as iPad.

[0088] (3) Portable entertainment devices: These devices can display and play multimedia content. This category of devices includes audio and video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.

[0089] (4) Server: A device that provides computing services, the server consists of a processor, a hard disk, a memory, a system bus, etc. The server is similar in architecture to a general-purpose computer, but due to the need to provide high-reliability services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.

[0090] (5) Embedded system: This system is installed in a robot system or an automated device to control the system or device it is in. This category of devices includes single-chip microcontrollers / microcontrollers (such as STM32 and ARM), single-board microcontrollers (such as Arduino and Raspberry Pi), and neural computing acceleration platforms (such as NVIDIA Jetson).

[0091] (6) Other electronic devices with data interaction functions.

[0092] The device embodiments described above are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs.

[0093] From the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, or by hardware. Based on this understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some part of the embodiment.

[0094] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task processing system, characterized in that, The task processing system can be applied to software- or hardware-based robots to achieve the robot's target task execution; the system includes: A biomimetic dynamic neural network model includes at least one network unit with multiple shared state neurons, each network unit including a behavioral neuron, wherein the state neurons and behavioral neurons can learn to form a behavioral-state relationship network. The biomimetic dynamic neural network model is able to perform state transitions and behavioral activity regulation based on the current state and the formed behavioral state relationship network when it receives input signals related to the target task. It can also control the corresponding downstream neural network model to regulate the robot used to perform the target task by regulating the output signals of behavioral neurons until the target task is completed.

2. The task processing system according to claim 1, characterized in that, Each network unit also includes a reward neuron; The system also includes at least one downstream neural network model; The biomimetic dynamic neural network model is connected to each downstream neural network model through behavioral neurons and reward neurons.

3. The task processing system according to claim 2, characterized in that, The behavioral state relationship network includes Lateral connections formed between state neurons to describe state transition paths; and Secondary connections are formed between the lateral connections and behavioral neurons; or include Lateral connections formed between state neurons to describe state transition paths; Secondary connections are formed between the lateral connections and behavioral neurons; and Connections are formed between state neurons and behavioral neurons.

4. The task processing system according to claim 3, characterized in that, When there are two or more network units, there is lateral inhibition between behavioral neurons in different network units.

5. The task processing system according to any one of claims 2 to 4, characterized in that, The reward neurons regulate the learning process within their respective network units based on internally generated or externally received incentive signals.

6. The task processing system according to claim 5, characterized in that, The behavioral neurons are capable of spontaneous excitation, and the learning includes exploratory learning resulting from the spontaneous excitation of the behavioral neurons.

7. The task processing system according to claim 6, characterized in that, Based on the stimulus signals received from the outside, state neurons in an excited state can form connections with reward neurons.

8. The task processing system according to claim 7, characterized in that, The biomimetic dynamic neural network model also includes target neurons for setting state goals, which can learn to form connections with specific state neurons.

9. The task processing system according to claim 8, characterized in that, The state neuron can carry a task potential, and the target neuron sets the state target by controlling the task potential of a specific state neuron connected to it. The neural network model can use the task potential and the formed behavioral state relationship network to search for a path from the current state to the state goal.

10. The task processing system according to claim 9, characterized in that, The neural network model can also utilize the task potentials of state neurons to form task currents of cross-state neurons in the behavioral state relationship network and the lateral inhibition existing between behavioral neurons to perform behavioral selection.

11. The task processing system according to any one of claims 1 to 10, characterized in that, The state neurons store state information using sparse coding, meaning that only a minority of state neurons are active at any given time. Here, "minority" means that the number of activated state neurons at any given time is no less than one and no more than half the total number of state neurons in each layer. The neural network also includes... A preprocessing unit used to transform and process the input signal to output a low-dimensional signal to the mapped neuron; and A mapping neuron that sparsely encodes an input low-dimensional signal, wherein the mapping neuron activates a corresponding state neuron by mapping the input.

12. The task processing system according to claim 11, characterized in that, The biomimetic dynamic neural network model also includes target neurons for setting state goals, which can learn to form connections with specific state neurons.

13. A robot, characterized in that, The robot is configured to execute a target task using a biomimetic dynamic neural network model when it receives the target task. The biomimetic dynamic neural network model includes at least one network unit with multiple shared state neurons. Each network unit includes a behavior neuron and a reward neuron. The state neurons and behavior neurons can learn to form a behavior-state relationship network. The biomimetic dynamic neural network model, upon receiving an input signal related to the target task, can perform state transitions and behavioral activity regulation based on the current state and the formed behavioral state relationship network. It can also control the corresponding downstream neural network model to regulate the robot's actions by regulating the output signals of behavioral neurons, so that the robot can complete the target task.

14. The robot according to claim 13, characterized in that, The biomimetic dynamic neural network model forms the behavioral state relationship network through the following learning method: The biomimetic dynamic neural network model is initialized so that connections are established between all state neurons and all reward neurons; It receives input signals, activates corresponding state neurons based on the input signals to learn, and forms a behavioral state relationship network, which includes: When no external stimulus signal is present, exploration and learning are carried out based on the spontaneous excitation of action neurons, and behavioral neurons are randomly activated. In network units where behavioral neurons are in an excited state, reward neurons are activated to release regulatory substances according to the excitation of state neurons. When an external reward signal is present, the reward neuron releases regulatory substances based on the received external reward signal; The modulatory substance is used to strengthen the lateral connections between the previously activated state neuron and the currently activated state neuron, as well as the secondary connections between these lateral connections and the behavioral neurons within the network unit, and to weaken the connections between the currently activated state neuron and the reward neuron.

15. The robot according to claim 14, characterized in that, The biomimetic dynamic neural network model further includes target neurons for setting state goals. These target neurons can learn to form connections with specific state neurons. The step of activating corresponding state neurons based on input signals to learn and form a behavioral state relationship network also includes... Depending on the presence or disappearance of external reward signals, the target neuron and reward neuron form or break connections with the current state neuron.

16. A robot having the task processing system according to any one of claims 1 to 12 deployed thereon.

Images