A brain-like pulse reinforcement evolution unmanned obstacle avoidance method and related device

Abstract

The application discloses an unmanned obstacle avoidance method based on brain-like pulse reinforcement evolution and related devices, and the application generates an obstacle avoidance action at the current time according to the state sequence of the unmanned vehicle at the current time and the cortex-basal ganglia-thalamus network, and optimizes the cortex-basal ganglia-thalamus network at the next time based on the obstacle avoidance action at the current time. In the optimization process of the cortex-basal ganglia-thalamus network, the alert value of the cortex-basal ganglia-thalamus network to the environment is accumulated according to the reward difference value at adjacent times. When the accumulated alert value does not exceed a second threshold, the cortex-basal ganglia-thalamus network is optimized based on a spike timing-dependent plasticity mechanism and a dopamine regulation mechanism in turn. When the accumulated alert value exceeds the second threshold, the cortex-basal ganglia-thalamus network is optimized based on an offline evolution method of a genetic algorithm, so that the obstacle avoidance method with fast adaptation to different complex environments and accurate decision-making is realized.

Description

TECHNICAL FIELD

[0001] The application relates to a brain-like spiking reinforcement evolution unmanned obstacle avoidance method and related device, and belongs to the unmanned driving field. BACKGROUND

[0002] The core of the unmanned driving technology is to realize autonomous decision and path planning of a vehicle in a complex dynamic environment, wherein the obstacle avoidance technology, as an important part of ensuring the safety and reliability of the unmanned vehicle, has always been a research focus and difficulty.

[0003] Although deep reinforcement learning, such as a deep Q-network (DQN), shows potential in policy learning, there is still a high demand for computing resources and training sample size, especially in dynamic and complex environments, and a long training time is required to achieve an optimal strategy.

[0004] A spiking neural network (SNN) is attracting attention due to its simulation of the pulse firing behavior of human brain neurons. Compared with traditional deep learning models, the SNN has higher computational efficiency and energy efficiency, and is considered as a potential solution to improve the real-time performance of the unmanned driving system. However, based on the spiking neural network, there is currently no obstacle avoidance method that can quickly adapt to different complex environments and make accurate decisions. SUMMARY

[0005] The application provides a brain-like spiking reinforcement evolution unmanned obstacle avoidance method and related device, which solves the problems disclosed in the background art.

[0006] According to one aspect of the present disclosure, a brain-like spiking reinforcement evolution unmanned obstacle avoidance method is provided, comprising:

[0007] obtaining a state sequence of an unmanned vehicle in a current time lane scene;

[0008] generating an obstacle avoidance action of the unmanned vehicle at the current time according to the state sequence and a spiking neural network; wherein the spiking neural network is a cortex-basal ganglia-thalamus network;

[0009] interacting the obstacle avoidance action with a current time environment to generate a reward given by the current time environment; wherein the current time environment is an environment in the current time lane scene;

[0010] if a difference between the reward at the current time and a reward at a previous time exceeds a first threshold value, copying the current time environment to an environment replay pool and calculating an alert value accumulated to the current time; wherein the alert value is an alert value of the spiking neural network to the environment;

[0011] If the accumulated vigilance value at the current time does not exceed the second threshold, the spiking neural network is optimized based on a spike-timing-dependent plasticity mechanism and a dopamine modulation mechanism in turn; wherein the optimized spiking neural network is used to generate the next time obstacle avoidance action of the unmanned vehicle;

[0012] If the accumulated vigilance value at the current time exceeds the second threshold, the spiking neural network is optimized based on an offline evolution method of genetic algorithm according to the current time environment in the environment replay pool.

[0013] Further, the spiking neuron A in the cortex-basal ganglia-thalamus network is constructed according to a gene regulation network A1 and an Izhikevich neuron model A2;

[0014] The Izhikevich neuron model A2 is used to model the firing behavior of the spiking neuron A;

[0015] The gene regulation network A1 is used to generate the synaptic weight between the spiking neuron A and the spiking neuron B, and the input of the gene regulation network A1 is the synaptic current of the input spiking neuron A; wherein the spiking neuron B is the spiking neuron receiving the output of the spiking neuron A.

[0016] Further, the reward given by the current time environment is generated, and the formula is:

[0017] ;

[0018] ;

[0019] ;

[0020] In the formula, r t is the reward given by the current time t environment, s t is the state sequence of the unmanned vehicle at the current time t, R b (s t ) is the total discrete reward corresponding to s t , Eva(s t ) is the continuous reward corresponding to s t , α is a fixed coefficient, d safe is the preset safety distance of the unmanned vehicle and the obstacle, d current is the distance between the unmanned vehicle and the obstacle at the current time, v current is the speed of the unmanned vehicle at the current time, R collision (s t ) is the discrete reward of the unmanned vehicle in the collision state with the obstacle, R speed (s t ) is the discrete reward calculated according to the interval in which the speed of the unmanned vehicle at the current time is located, and R obstacle (st ) is a discrete reward for whether the autonomous vehicle keeps a safe distance from the obstacle, R lane-change (s t ) is a discrete reward for whether the autonomous vehicle successfully changes lanes to avoid the obstacle, R fre-change (s t ) is a discrete reward for whether the autonomous vehicle continuously changes lanes, y1 and y2 are preset values, and C1-C4 are four different states in which the autonomous vehicle is located.

[0021] Further, the vigilance value accumulated to the current time is calculated, and the formula is:

[0022] ;

[0023] In the formula, is the vigilance value accumulated to the current time t, β is the vigilance value accumulated to the last time, μ is the update rate of δ(t), and δ(t) is the difference between the reward at the current time and the reward at the last time.

[0024] Further, the pulse neural network is optimized based on the spike-timing-dependent plasticity mechanism and the dopamine regulation mechanism in turn, including:

[0025] The pulse neural network is optimized for the first time based on the spike-timing-dependent plasticity mechanism, wherein the first optimization is to optimize the synaptic weights between the dorsolateral prefrontal cortex and the D1 type dopamine receptors in the striatum and the synaptic weights between the dorsolateral prefrontal cortex and the D2 type dopamine receptors in the striatum in the pulse neural network.

[0026] If the difference between the reward at the current time and the reward at the last time is greater than 0, on the basis of the first optimization, the synaptic weights between the dorsolateral prefrontal cortex and the D1 type dopamine receptors in the striatum are increased, and the synaptic weights between the dorsolateral prefrontal cortex and the D2 type dopamine receptors in the striatum are decreased.

[0027] If the difference between the reward at the current time and the reward at the last time is not greater than 0, on the basis of the first optimization, the synaptic weights between the dorsolateral prefrontal cortex and the D1 type dopamine receptors in the striatum are decreased, and the synaptic weights between the dorsolateral prefrontal cortex and the D2 type dopamine receptors in the striatum are increased.

[0028] Further, according to the current time environment in the environment replay pool, the pulse neural network is optimized based on an offline evolution method of a genetic algorithm, including:

[0029] S1) Obtain the pulse neurons of the dorsolateral prefrontal cortex of the pulse neural network to be optimized, and take the obtained pulse neurons as initial nuclei;

[0030] S2) randomly expanding the initial core group to obtain a plurality of new core groups, replacing the initial core group in the to-be-optimized spiking neural network with the new core group to obtain a plurality of new to-be-optimized spiking neural networks;

[0031] S3) traversing all the to-be-optimized spiking neural networks to obtain a reward corresponding to the elite network and a reward corresponding to the to-be-optimized spiking neural network in the current time environment, and taking a difference between the reward corresponding to the initial elite network and the reward corresponding to the to-be-optimized spiking neural network as a reward of the corresponding core group, and calculating a reward of each spiking neuron in the core group according to the reward of the core group and the synaptic weight of each spiking neuron in the core group; wherein the initial elite network is a spiking neural network with the highest reward in the historical time;

[0032] S4) if the number of iterations reaches a third threshold, taking the first N spiking neurons with high rewards as the final core group to obtain an optimized spiking neural network, and if the number of iterations does not reach the third threshold, performing mutation and breeding operations on the first N spiking neurons with high rewards, taking the spiking neurons after the mutation and breeding operations as the initial core group, and turning to S2); wherein N is the number of dorsolateral prefrontal cortex spiking neurons of the spiking neural network that is not optimized by the offline evolution method.

[0033] According to another aspect of the present disclosure, an unmanned obstacle avoidance device based on brain-like spiking reinforcement evolution is provided, comprising:

[0034] An acquisition module acquires a state sequence of an unmanned vehicle in a current time lane scene;

[0035] An obstacle avoidance action generation module generates an obstacle avoidance action of the unmanned vehicle in the current time according to the state sequence and a spiking neural network; wherein the spiking neural network is a cortex-basal ganglia-thalamus network;

[0036] A reward generation module generates a reward given by the current time environment by interacting the obstacle avoidance action with the current time environment; wherein the current time environment is the environment in the current time lane scene;

[0037] A vigilance value accumulation module copies the current time environment to an environment replay pool and calculates the vigilance value accumulated to the current time if a difference between the reward in the current time and the reward in the last time exceeds a first threshold; wherein the vigilance value is the vigilance value of the spiking neural network to the environment;

[0038] A first optimization module optimizes the spiking neural network based on a spike-timing-dependent plasticity mechanism and a dopamine regulation mechanism in turn if the vigilance value accumulated to the current time does not exceed a second threshold; wherein the optimized spiking neural network is used to generate an obstacle avoidance action of the unmanned vehicle in the next time;

[0039] The second optimization module is configured to, if the accumulated vigilance value at the current time exceeds a second threshold value, optimize the spiking neural network based on an offline evolution method of a genetic algorithm according to the current time environment in the environment replay pool.

[0040] Further, the reward generation module is configured to generate a reward given by the current time environment, and the formula is:

[0041] ;

[0042] ;

[0043] ;

[0044] In the formula, r t is the reward given by the current time environment t, s t is a state sequence of the unmanned vehicle at the current time t, R b (s t ) is a total discrete reward corresponding to s t , Eva(s t ) is a continuous reward corresponding to s t , a is a fixed coefficient, d safe is a preset safety distance of the unmanned vehicle from the obstacle, d current is a distance of the unmanned vehicle from the obstacle at the current time, v current is a speed of the unmanned vehicle at the current time, R collision (s t ) is a discrete reward of the unmanned vehicle in a collision state with the obstacle, R speed (s t ) is a discrete reward calculated according to a speed interval of the unmanned vehicle at the current time, R obstacle (s t ) is a discrete reward of whether the unmanned vehicle maintains a safety distance from the obstacle, R lane-change (s t ) is a discrete reward of the unmanned vehicle successfully changing lanes to avoid the obstacle, R fre-change (s t ) is a discrete reward of the unmanned vehicle continuously changing lanes, y1 and y2 are preset values, and C1-C4 are four different states of the unmanned vehicle.

[0045] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores one or more programs including instructions, which, when executed by a computing device, cause the computing device to perform the brain-inspired spiking reinforcement evolution-based unmanned obstacle avoidance method.

[0046] According to another aspect of the present disclosure, a computer device is provided, comprising one or more processors, and one or more memories, one or more programs stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the unmanned obstacle avoidance method based on the brain-like pulse reinforcement evolution.

[0047] The present application has the following beneficial effects: the present application generates the current time obstacle avoidance action according to the state sequence of the unmanned vehicle at the current time and the cortex-basal ganglia-thalamus network, and optimizes the cortex-basal ganglia-thalamus network at the next time based on the current time obstacle avoidance action. In the optimization process of the cortex-basal ganglia-thalamus network, the alert value of the cortex-basal ganglia-thalamus network to the environment is accumulated according to the reward difference of adjacent time. When the accumulated alert value does not exceed the second threshold, the cortex-basal ganglia-thalamus network is optimized based on the spike timing-dependent plasticity mechanism and the dopamine regulation mechanism in turn. When the accumulated alert value exceeds the second threshold, the cortex-basal ganglia-thalamus network is optimized based on the offline evolution method of genetic algorithm. The rapid adaptation and decision accuracy of the cortex-basal ganglia-thalamus network to the complex environment can be effectively improved, so as to realize the obstacle avoidance method with rapid adaptation and accurate decision to different complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the brain-like pulse reinforcement evolution unmanned obstacle avoidance method;

[0049] Figure 2 The schematic diagram of the gene regulation network;

[0050] Figure 3 The structural schematic diagram of the cortex-basal ganglia-thalamus network;

[0051] Figure 4 The total schematic diagram of the optimized pulse neural network;

[0052] Figure 5 The flowchart of the pulse neural network optimized based on the spike timing-dependent plasticity mechanism and the dopamine regulation mechanism in turn;

[0053] Figure 6 The flowchart of the pulse neural network optimized based on the offline evolution method of genetic algorithm according to the current time environment in the environment replay pool;

[0054] Figure 7 The block diagram of the brain-like pulse reinforcement evolution unmanned obstacle avoidance device. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present disclosure will be described clearly and completely in the embodiments of the present disclosure in combination with the drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one example embodiment is merely illustrative in nature and in no way limits the present disclosure and its application or uses. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present disclosure.

[0056] Unless specifically stated otherwise, the relative arrangements of the components and steps, numerical expressions, and values shown in the embodiments presented herein are not meant to limit the scope of the present disclosure.

[0057] It should be understood, however, that the sizes of the various portions shown in the drawings are not necessarily to scale with one another.

[0058] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0059] In all of the examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of example embodiments can have different values.

[0060] It should be noted that like symbols and letters in the drawings represent like items, such that once an item is defined in one drawing, it is not necessarily discussed further in subsequent drawings.

[0061] Meanwhile, in the description of the present disclosure, the terms "first", "second", and the like are used only to distinguish the description, and cannot be understood as indicating or implying relative importance. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0062] Spiking neural network is a new generation of artificial neural network model inspired by biology, which belongs to the subset of deep learning and has strong biological basis support. Spiking neural network contains neuron nodes with time series dynamics, synapse structure with steady-state plasticity balance, network loop with specific functions, etc., which highly draws on the biological optimization methods of local unsupervised (such as pulse time-dependent plasticity, short-term synaptic plasticity, local steady-state regulation, etc.), global weak supervision (such as dopamine reward learning, energy-based function optimization, etc.) inspired by biology, and thus has strong ability of spatio-temporal information representation, asynchronous event information processing, network self-organizing learning, etc.

[0063] This invention provides an obstacle avoidance method for autonomous driving based on spiking neural networks. The aim is to construct and optimize a spiking neural network capable of rapid adaptation and accurate decision-making in various complex environments, thereby enabling obstacle avoidance in autonomous driving, which is the core of autonomous driving technology. This obstacle avoidance method can be executed by an obstacle avoidance device, which can be a terminal device or a server. The terminal device can include, but is not limited to, computers, intelligent vehicle-mounted devices, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, big data, and artificial intelligence platforms, etc., etc., without limitation. Optionally, this obstacle avoidance method can also be executed collaboratively by multiple electronic devices with computing power. For ease of explanation, subsequent embodiments will be described as being executed by an obstacle avoidance device.

[0064] Please see Figure 1 , Figure 1 This is a flowchart of the autonomous driving obstacle avoidance method based on brain-like impulse enhancement evolution provided by the present invention. This autonomous driving obstacle avoidance method can be executed by an obstacle avoidance device and may include at least the following steps:

[0065] like Figure 1 As shown, in step 1 of the embodiment, the state sequence of the autonomous vehicle in the lane scene at the current moment is obtained.

[0066] It should be noted that the lane scene refers to the scene of autonomous vehicles driving on the road. The lane scene can be collected by the camera of the autonomous vehicle. Furthermore, through existing video image recognition technology, the environmental information of the autonomous vehicle can be obtained, such as whether there are cars within 100 meters of the autonomous vehicle, the distance of other vehicles to itself, etc. Combined with the position, speed, acceleration and other data collected by the autonomous vehicle's onboard system, they can be used to form the state sequence of the autonomous vehicle.

[0067] Suppose that the state information of the autonomous vehicle in lane t at the current time t is (S1, S2, ..., S...). n ), S1~S n This represents n state information points, which may include the autonomous vehicle's current position, speed, acceleration, whether there are other vehicles within a range (e.g., 100 meters), and the distance between other vehicles and the autonomous vehicle. To facilitate subsequent processing, this state information needs to be preprocessed. Specifically, continuous values ​​in the state information can be linearly normalized, and the normalized data can be encoded. The encoding formula can be expressed as Encode(S) i This allows us to obtain the set of state pulse sequences at the current time t. ;in, S i is the ith state information of the unmanned vehicle, i∈[1, n], and Encode() is an encoding function by which the state pulse sequence can be obtained.

[0068] Return Figure 1 , step 2 of the embodiment, generating an obstacle avoidance action of the unmanned vehicle at the current time according to the state sequence and the pulse neural network; wherein the pulse neural network is a cortex-basal ganglia-thalamus network.

[0069] It should be noted that a simple pulse neuron only has the ability to calculate a forward pulse signal and does not have an evolutionary function in a biological sense, so in some embodiments, the pulse neuron in the cortex-basal ganglia-thalamus network is an evolutionary pulse neuron (hereinafter referred to as “pulse neuron” in the following). Such a pulse neuron has the ability to evolve through a gene regulation network, which is conducive to the optimization of the pulse neuron by subsequent evolutionary calculation.

[0070] Taking pulse neuron A as an example, pulse neuron A is constructed according to gene regulation network A1 and Izhikevich neuron model A2; wherein Izhikevich neuron model A2 is used to model the firing behavior of pulse neuron A; gene regulation network A1 is used to generate the synaptic weight between pulse neuron A and pulse neuron B, and the input of gene regulation network A1 is the synaptic current of input pulse neuron A; wherein pulse neuron B is the pulse neuron receiving the output of pulse neuron A. Similarly, taking pulse neuron B as an example, it is constructed according to gene regulation network B1 and Izhikevich neuron model B2.

[0071] The above Izhikevich neuron model is an existing neuron model (E. M. Izhikevich, “Simple model of spiking neurons,” IEEE Transactions on neural networks, vol. 14, no. 6, pp. 1569-1572, 2003), which is used to model the firing behavior of the pulse neuron, describe the dynamic changes of the membrane potential and the recovery variable of the pulse neuron, and can be expressed by the formula:

[0072] ;

[0073] ;

[0074] In the formula, I is the synaptic current input of the pulse neuron, v and u are the membrane potential and the recovery variable of the pulse neuron, respectively, and a and b are parameters for controlling the speed and recovery characteristics of the Izhikevich neuron model, respectively.

[0075] The maximum voltage of the spiking neuron can be set to 30mV. The voltage resets when the spiking neuron fires at 30mV. This can be expressed by the formula:

[0076] ;

[0077] In the formula, c is the reset amount and d is the increment amount, indicating that when v is greater than or equal to 30mv, the value is reset, that is, c is assigned to v, and d is added to the original u.

[0078] It should be noted that the gene regulatory network is a gene regulatory network based on artificial genome coding; see [link to relevant documentation]. Figure 2 An artificial genome, based on a fixed number of genes, is constructed specifically of promoters, regulatory regions, and gene-coding regions. The promoter region's main function is to initially process input signals and provide an initial activity level for gene expression. The regulatory region is the intermediate layer of gene expression, responsible for integrating and adjusting signals based on input signals and gene expression values. The gene-coding region is responsible for the actual gene expression calculations, and its output is used to calculate synaptic weights between spiking neurons. The gene regulatory network includes input nodes, regulatory nodes (… Figure 2 The solid lines represent primary regulatory nodes, and the dashed lines represent secondary regulatory nodes to be developed, as well as output nodes. Input nodes (corresponding to promoters) determine the initial activity of gene expression, while regulatory nodes enable the regulation between gene expressions. The synaptic current of the input spiking neuron serves as the input to the gene regulatory network, and the output is the synaptic weight connected to the current spiking neuron. The synaptic weight is jointly determined by the synaptic current, promoter, and regulatory region.

[0079] Gene expression is influenced by promoters, regulatory matrices, and synaptic currents. Regulatory effect values ​​are calculated using the regulatory matrix and synaptic currents, and gene expression is calculated based on the promoter. Gene expression values ​​are then normalized and updated. Synaptic weights are obtained by mapping these gene expression values, which can be expressed by the following formula:

[0080] V = R × I;

[0081] ΔE = P × V;

[0082] ΔE norm =(ΔE-μ ΔE ) / σ ΔE ;

[0083] W=tanh(ΔE norm );

[0084] In the formula, V is the regulatory effect value, R is the regulatory matrix, tanh() is the hyperbolic tangent function, ΔE is the initial (unnormalized) gene expression value, P is the promoter influence matrix, and ΔE normwhere μ is the mean of all gene expression values, and σ is the standard deviation of all gene expression values. ΔE where μ is the mean of all gene expression values, and σ is the standard deviation of all gene expression values. ΔE where μ is the mean of all gene expression values, and σ is the standard deviation of all gene expression values. norm where μ is the mean of all gene expression values, and σ is the standard deviation of all gene expression values.

[0085] It should be noted that the cortical-basal ganglia-thalamic network is a hybrid model that combines reinforcement learning and biological inspiration to mimic the cortical-basal ganglia-thalamic structure of the brain.

[0086] Referring to Figure 3 The cortical-basal ganglia-thalamic network includes the dorsolateral prefrontal cortex, the orbitofrontal cortex, the substantia nigra pars compacta / ventral tegmental area, the striatal D1 dopamine receptor, the striatal D2 dopamine receptor, the subthalamic nucleus, the lateral globus pallidus, the reticulata / substantia nigra pars reticulata, the thalamus, and the primary motor area. The output of the substantia nigra pars compacta / ventral tegmental area is excitatory connected to the input of the orbitofrontal cortex, the output of the dorsolateral prefrontal cortex is excitatory connected to the input of the striatal D1 dopamine receptor, the input of the striatal D2 dopamine receptor, and the input of the subthalamic nucleus, respectively, the orbitofrontal cortex excitatory influences the excitatory connection of the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor, the excitatory connection of the dorsolateral prefrontal cortex and the subthalamic nucleus, the output of the striatal D1 dopamine receptor is inhibitory connected to the input of the reticulata / substantia nigra pars reticulata, the output of the striatal D2 dopamine receptor is inhibitory connected to the input of the lateral globus pallidus, the output of the lateral globus pallidus is excitatory connected to the input of the reticulata / substantia nigra pars reticulata, the output of the subthalamic nucleus is excitatory connected to the input of the reticulata / substantia nigra pars reticulata, the output of the subthalamic nucleus is excitatory connected to the input of the lateral globus pallidus, the output of the lateral globus pallidus is inhibitory connected to the input of the subthalamic nucleus, the output of the reticulata / substantia nigra pars reticulata is excitatory connected to the input of the thalamus, the output of the thalamus is excitatory connected to the input of the primary motor area, and the output of the dorsolateral prefrontal cortex is excitatory connected to the input of the primary motor area.

[0087] In the cortical-basal ganglia-thalamic network, the dorsolateral prefrontal cortex receives the state pulse sequence and transmits it to the basal ganglia, which includes a direct pathway, an indirect pathway and a hyper-direct pathway. The direct pathway includes the striatum D1 dopamine receptor and the substantia nigra pars reticulata / medial part of globus pallidus, the striatum D1 dopamine receptor directly inhibits the substantia nigra pars reticulata / medial part of globus pallidus. The indirect pathway includes the striatum D2 dopamine receptor, the lateral part of globus pallidus and the substantia nigra pars reticulata / medial part of globus pallidus. The D2 dopamine receptor inhibits the lateral part of globus pallidus, thereby inhibiting the substantia nigra pars reticulata / medial part of globus pallidus. The hyper-direct pathway includes the subthalamic nucleus, the lateral part of globus pallidus and the substantia nigra pars reticulata / medial part of globus pallidus. The dorsolateral prefrontal cortex activates the subthalamic nucleus, which further excites the lateral part of globus pallidus and the substantia nigra pars reticulata / medial part of globus pallidus. The three pathways converge at the substantia nigra pars reticulata / medial part of globus pallidus, which is the output area of the basal ganglia, regulates the inhibitory output of the substantia nigra pars reticulata / medial part of globus pallidus to the thalamus, and then integrates the excitatory input of the dorsolateral prefrontal cortex to affect the excitatory output of the thalamus to the primary motor area, i.e. the substantia nigra pars reticulata / medial part of globus pallidus generates obstacle avoidance action selection, the thalamus receives the projection of the substantia nigra pars reticulata / medial part of globus pallidus and transmits the obstacle avoidance action to the primary motor area and outputs the obstacle avoidance action.

[0088] The above-mentioned cortical-basal ganglia-thalamic network considers the signal response mechanism of the orbital cortex. The orbitofrontal cortex in the orbital cortex receives the difference in reward between adjacent time points, i.e. the difference in reward between adjacent time points is received through the substantia nigra compacta / ventral tegmental area.

[0089] The timing difference error has a top-down biasing effect on the basal ganglia. In the basal ganglia, the direct pathway and the hyper-direct pathway are responsible for generating obstacle avoidance actions, and the indirect pathway is an inhibitor that suppresses the generation of erroneous obstacle avoidance actions.

[0090] Return Figure 1 In step 3 of the embodiment, the obstacle avoidance action is interacted with the current time environment to generate the reward given by the current time environment; wherein the current time environment is the environment in the current time lane scene.

[0091] It should be noted that in some embodiments, the reward is divided into two types, one is discrete reward and the other is continuous reward. The individual continuous reward cannot accurately represent the reward information from the environment. Considering the discreteness of the state, both discrete reward and continuous reward are set, which is conducive to better representing the reward based on environmental feedback, while ensuring the stability of the reward value.

[0092] If the ego vehicle collides with an obstacle (specifically, a stationary obstacle), a very large negative reward should be given to strongly penalize this unsafe behavior; to encourage the ego vehicle to maintain a proper speed while driving, a reward can be given according to the speed; to encourage the ego vehicle to actively avoid obstacles, a positive reward can be given in each time step if the ego vehicle maintains a sufficient safety distance from the obstacle in front; if the car successfully changes lanes to avoid an obstacle, a positive reward can be given to encourage it to learn to change lanes in appropriate situations, however, to avoid the car changing lanes frequently leading to unnecessary risks, a small negative reward can be set to penalize unnecessary lane changes.

[0093] Therefore, the mathematical formula of the discrete reward can be expressed as:

[0094] R collision (s t )=y2;

[0095] ;

[0096] ;

[0097] R lane-change (s t )=y6;

[0098] R fre-change (s t )=y4;

[0099] ;

[0100] In the formula, R b (s t ) is the total discrete reward corresponding to s t , s t is the state sequence of the ego vehicle at the current time t, R collision (s t ) is the discrete reward of the ego vehicle being in a collision state with an obstacle, R speed (s t ) is the discrete reward calculated according to the interval in which the speed of the ego vehicle at the current time is located, R obstacle (s t ) is the discrete reward of whether the ego vehicle maintains a safe distance from the obstacle, R lane-change (s t ) is the discrete reward of the ego vehicle successfully changing lanes to avoid an obstacle, and R fre-change (s t) is the discrete reward of the continuous lane change of the unmanned vehicle, y1~y6 are preset values, which can be 10, -1000, 5, -5, -10, 20 respectively, and the values can be determined according to actual conditions, v min and v max are respectively the lower limit and the upper limit of the speed of the unmanned vehicle, which can be set as 5 km / h and 50 km / h respectively, D safe is the preset safe distance between the unmanned vehicle and the obstacle, which can be set as 20 meters.

[0101] Based on the continuous state sequence of the unmanned vehicle, the state is divided into four ranges of discrete states, and C1~C4 are defined as four different states of the unmanned vehicle; C1 is that the distance between the unmanned vehicle and the obstacle is less than the safe distance, and a negative reward is given to this state; C2 is that there is an obstacle within the range of the unmanned vehicle (i.e. the sensor range, such as 100 meters), and the relative position of the unmanned vehicle relative to the obstacle needs to be evaluated; C3 is that when the unmanned vehicle maintains a safe distance and there is no particular danger, the reward is y1 (i.e. 10), indicating that the current state is neutral and there is no need for special reward or punishment; C4 is that if the unmanned vehicle collides with the obstacle, a very large negative reward is given to this state.

[0102] Therefore, the mathematical formula of the continuous reward can be expressed as:

[0103] ;

[0104] ;

[0105] In the formula, r t is the reward given by the environment at the current time t, Eva(s t ) is the corresponding continuous reward of s t , α is a fixed coefficient, d safe is the preset safe distance between the unmanned vehicle and the obstacle, d current is the distance between the unmanned vehicle and the obstacle at the current time, v current is the speed of the unmanned vehicle at the current time.

[0106] Returning to Figure 1 , in step 4 of the embodiment, if the difference between the reward at the current time and the reward at the last time exceeds the first threshold value, the environment at the current time is copied to the environment replay pool, and the vigilance value accumulated to the current time is calculated; wherein the vigilance value is the vigilance value of the spiking neural network to the environment.

[0107] It should be noted that the reward at the current time is transmitted to the substantia nigra pars compacta / ventral tegmental area, the difference between the reward at the current time and the reward at the last time is calculated, and the difference is defined as the reinforcement signal δ t , and the formula can be expressed as:

[0108] δ(t) = r t -r t-1 ;

[0109] wherein, δ(t) is the difference between the reward of the current moment and the reward of the last moment, r t-1 is the reward given by the environment at the last moment t-1.

[0110] The anterior cingulate cortex (anterior cingulate cortex model outside the cortical-basal ganglia-thalamic network) judges whether the reinforcement signal exceeds the first threshold, which can be set to 10.

[0111] When δ t does not exceed the first threshold, it means that the spiking neural network does not need to be vigilant about the current moment environment, i.e. there is no vigilance value at the current moment, and there is no need to copy the current moment environment to the environment replay pool, and the accumulated vigilance value remains unchanged.

[0112] Referring to Figure 4 , when δ t exceeds the first threshold, it means that the spiking neural network needs to be vigilant about the current moment environment, i.e. it is necessary to copy the current moment environment to the environment replay pool, and it is necessary to calculate the current moment vigilance value and accumulate the vigilance value, which can be expressed by the formula:

[0113] ;

[0114] wherein, is the accumulated vigilance value to the current moment t, β is the accumulated vigilance value to the last moment, μ is the update rate of δ(t), δ(t) is the difference between the reward of the current moment and the reward of the last moment, and μ is the update rate, which can be set to 0.5.

[0115] Based on the vigilance threshold mechanism of the anterior cingulate cortex, an environment replay pool is constructed, and the evolved optimization spiking neural network using genetic algorithm can be used for the vigilance environment, so that a larger strategy space can be explored and a better unmanned obstacle avoidance strategy can be obtained.

[0116] Returning to Figure 1 , step 5 of the embodiment, if the accumulated vigilance value to the current moment does not exceed the second threshold, the spiking neural network is optimized based on the spike-timing-dependent plasticity mechanism and the dopamine regulation mechanism in turn; if the accumulated vigilance value to the current moment exceeds the second threshold, the spiking neural network is optimized based on the off-line evolution method of genetic algorithm according to the current moment environment in the environment replay pool; wherein the optimized spiking neural network is used to generate the obstacle avoidance action of the unmanned vehicle at the next moment.

[0117] Referring to Figure 4When the accumulated alert value does not exceed the second threshold (which can be set to 100), the pulse neural network is optimized based on the spike-timing-dependent plasticity mechanism and the dopamine regulation mechanism in turn, specifically, the synaptic weights between the dorsolateral prefrontal cortex pulse neuron and the striatum D1 dopamine receptor pulse neuron are optimized, and the synaptic connection weights between the dorsolateral prefrontal cortex pulse neuron and the striatum D2 dopamine receptor pulse neuron are optimized.

[0118] In some embodiments, referring to Figure 5 The above optimization method can include:

[0119] S51) performing first optimization on the optimized pulse neural network based on the spike-timing-dependent plasticity mechanism; wherein the first optimization is to optimize the synaptic weights between the dorsolateral prefrontal cortex pulse neuron and the striatum D1 dopamine receptor pulse neuron, and to optimize the synaptic connection weights between the dorsolateral prefrontal cortex pulse neuron and the striatum D2 dopamine receptor pulse neuron.

[0120] It should be noted that, by using the spike-timing-dependent plasticity mechanism, if the spike of the pre-pulse neuron occurs before the spike of the post-pulse neuron, the synaptic weight will be enhanced, and if the spike of the pre-pulse neuron occurs after the spike of the post-pulse neuron, the synaptic weight will be weakened; which can be expressed by the formula:

[0121] ;

[0122] In the formula, Δw is the update amount of the synaptic weight, A + is a proportional factor of the amplitude constant of the synaptic weight increase, A - is a proportional factor of the amplitude constant of the synaptic weight decrease, Δt is the time difference between the pre-pulse neuron and the post-pulse neuron, specifically, the time interval between the signal emission of the pre-pulse neuron and the signal emission of the post-pulse neuron, is a time constant, which determines the decay speed of the inhibitory effect over time.

[0123] S52) judging whether the difference between the reward at the current time and the reward at the last time is greater than 0, if yes, turning to S53), if not, turning to S54).

[0124] S53) on the basis of the first optimization, increasing the synaptic weight between the dorsolateral prefrontal cortex and the striatum D1 dopamine receptor, and decreasing the synaptic weight between the dorsolateral prefrontal cortex and the striatum D2 dopamine receptor.

[0125] S54) If the difference between the reward at the current time and the reward at the last time is not greater than 0, on the basis of the first optimization, the synaptic weight between the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor is reduced, and the synaptic weight between the dorsolateral prefrontal cortex and the striatal D2 dopamine receptor is increased.

[0126] The synaptic weight increase and decrease can be expressed by the formula as follows:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] In the formula, DA D1 is the increase / decrease coefficient of the synaptic weight between the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor, DA D2 is the increase / decrease coefficient of the synaptic weight between the dorsolateral prefrontal cortex and the striatal D2 dopamine receptor, and are the synaptic weights between the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor after and before the increase / decrease, respectively, and are the synaptic weights between the dorsolateral prefrontal cortex and the striatal D2 dopamine receptor after and before the increase / decrease, respectively.

[0132] The spike-time-dependent plasticity mechanism is a commonly used weight update mechanism of the spiking neural network, which is similar to the back propagation mechanism in the neural network, and can effectively amplify the difference between the direct pathway and the indirect pathway in combination with the dopamine regulation mechanism, and is helpful for clear action selection.

[0133] Referring to Figure 4 , when the accumulated vigilance value exceeds the second threshold value, the spiking neural network is optimized based on an offline evolution method of a genetic algorithm according to the current time environment in the environment playback pool, and a niche sharing mechanism is adopted in the offline evolution method, so that the spiking neural network can obtain more accurate rewards.

[0134] In some embodiments, referring to Figure 6 , the optimization method can include:

[0135] S1) Obtain the pulse neuron of the dorsolateral prefrontal cortex of the spiking neural network to be optimized, and take the obtained pulse neuron as an initial kernel.

[0136] S2) randomly expanding the initial kernel to obtain a plurality of new kernels, replacing the initial kernel in the to-be-optimized spiking neural network with the new kernel to obtain a plurality of new to-be-optimized spiking neural networks.

[0137] In the above step, the random expansion is to randomly modify the synaptic weights of the spiking neurons in the initial kernel, thereby expanding a plurality of new kernels. The above two steps are equivalent to population initialization, randomly initializing the spiking neurons of the dorsolateral prefrontal cortex as the population, each spiking neuron containing a promoter and a regulatory matrix, the promoter determining the activation rule of the spiking neuron, and the regulatory matrix describing the distribution and dynamic change of the synaptic weights.

[0138] S3) traversing all the to-be-optimized spiking neural networks, obtaining the reward corresponding to the elite network and the reward corresponding to the to-be-optimized spiking neural network in the current environment, and taking the difference between the reward corresponding to the initial elite network and the reward corresponding to the to-be-optimized spiking neural network as the reward of the corresponding kernel, and calculating the reward of each spiking neuron in the kernel according to the reward of the kernel and the synaptic weight of each spiking neuron in the kernel; wherein the initial elite network is the spiking neural network with the highest reward at the historical moment.

[0139] The elite network and the to-be-optimized spiking neural network are interacted in the replication environment (i.e. the replicated current environment), and the rewards of the two are obtained, and the difference between the two rewards is taken as the reward of the corresponding kernel. For example, the to-be-optimized spiking neural network is the to-be-optimized spiking neural network of the corresponding kernel H, and the difference is the reward of the kernel H.

[0140] A niche sharing mechanism based on synaptic weights of spiking neurons is introduced, and the overall reward of the kernel H is distributed to each spiking neuron in the kernel H. Specifically, the reward of each spiking neuron is obtained according to the reward of the kernel H and the synaptic weight of each spiking neuron, so as to ensure the adaptability and reasonable evaluation of the reward of different spiking neurons in the competition.

[0141] For the spiking neuron, its activation pattern can be directly defined as the response to the input stimulus (i.e. the generated pulse output), and the pulse train of each spiking neuron can be represented by a vector to record the pulse occurrence in the time window. When distributing the reward, the functional overlap of the spiking neurons can be further considered. The functional overlap degree between the spiking neurons is measured by using the activation pattern (i.e. the response to the input stimulus) of the spiking neurons, and the reward sharing is performed on the highly overlapped spiking neurons based on the inner product when distributing the reward, so as to reduce the homogeneous competition.

[0142] The above niche sharing mechanism can be expressed by the following formula:

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] wherein, is the reward of the jth spiking neuron, w j is the synaptic weight of the jth spiking neuron, w k is the synaptic weight of the kth spiking neuron, is a parameter to adjust the strength of reward allocation, , R LPFC is the reward of the cluster, is the functional overlap (measured by the activation pattern between spiking neurons) of the jth and kth spiking neurons, are the activation sequences of the jth and kth spiking neurons at the current time t, respectively.

[0148] S4) determining whether the iteration number reaches a third threshold (which can be set as 1000), if yes, turning to S5), if not, turning to S6.

[0149] S5) taking the top N spiking neurons with high reward as the final cluster to obtain the optimized spiking neural network; wherein, N is the number of the dorsolateral prefrontal cortex spiking neurons of the spiking neural network which is not optimized by the offline evolution method, and assuming that the number of the dorsolateral prefrontal cortex spiking neurons before optimization is 4422, then N here is also 4422.

[0150] S6) performing mutation and breeding operations on the top N spiking neurons with high reward, and taking the spiking neurons after mutation and breeding operations as the initial cluster, and turning to S2).

[0151] It should be noted that the mutation operation includes changing the initial activity of the gene by starting the mutation of the gene, and adjusting the synaptic weight or dynamic change mode by mutating the regulatory region gene. Through mutation and breeding operations, new spiking neurons are generated to form the next generation population.

[0152] The method generates an obstacle avoidance action at the current moment according to a state sequence of the unmanned vehicle at the current moment and the cortex-basal ganglia-thalamus network, and optimizes the cortex-basal ganglia-thalamus network at the next moment based on the obstacle avoidance action at the current moment. In the optimization process of the cortex-basal ganglia-thalamus network, the vigilance value of the cortex-basal ganglia-thalamus network to the environment is accumulated according to a reward difference value at adjacent moments. When the accumulated vigilance value does not exceed a second threshold, the cortex-basal ganglia-thalamus network is optimized based on a spike-timing-dependent plasticity mechanism and a dopamine regulation mechanism in turn. When the accumulated vigilance value exceeds the second threshold, the cortex-basal ganglia-thalamus network is optimized based on an offline evolution method of a genetic algorithm. The method can effectively improve the rapid adaptation and decision accuracy of the cortex-basal ganglia-thalamus network to a complex environment, thereby realizing an obstacle avoidance method that rapidly adapts to different complex environments and is accurate in decision-making.

[0153] Referring to Figure 7 An unmanned obstacle avoidance device of a brain-like pulse reinforcement evolution is provided for the present disclosure, Figure 7 Embodiments of the present disclosure are virtual devices that can be loaded and executed by a computer device, which can include the above-mentioned obstacle avoidance device, Figure 7 The brain-like pulse reinforcement evolution unmanned obstacle avoidance device shown can include an acquisition module, an obstacle avoidance action generation module, a reward generation module, a vigilance value accumulation module, a first optimization module, and a second optimization module, which are used to execute the above-mentioned brain-like pulse reinforcement evolution unmanned obstacle avoidance method, and can:

[0154] The acquisition module is configured to acquire a state sequence of an unmanned vehicle in a lane scene at a current moment.

[0155] The obstacle avoidance action generation module is configured to generate an obstacle avoidance action of the unmanned vehicle at the current moment according to the state sequence and a pulse neural network; wherein the pulse neural network is a cortex-basal ganglia-thalamus network.

[0156] The reward generation module is configured to interact the obstacle avoidance action with a current moment environment to generate a reward given by the current moment environment; wherein the current moment environment is an environment in the lane scene at the current moment.

[0157] In the reward generation module, the reward given by the current moment environment is generated, and the formula is:

[0158] ;

[0159] ;

[0160] ;

[0161] r ta reward given to the environment at the current time t, s t a state sequence of the unmanned vehicle at the current time t, R b (s t ) a discrete reward of s t a corresponding total discrete reward, Eva(s t ) a continuous reward of s t , a fixed coefficient, d safe a preset safety distance between the unmanned vehicle and the obstacle, d current a distance between the unmanned vehicle and the obstacle at the current time, v current a speed of the unmanned vehicle at the current time, R collision (s t ) a discrete reward of the unmanned vehicle being in a collision state with the obstacle, R speed (s t ) a discrete reward calculated according to an interval in which the speed of the unmanned vehicle at the current time is located, R obstacle (s t ) a discrete reward of the unmanned vehicle whether to keep a safety distance from the obstacle, R lane-change (s t ) a discrete reward of the unmanned vehicle successfully changing lanes to avoid the obstacle, R fre-change (s t ) a discrete reward of the unmanned vehicle continuously changing lanes, y1 and y2 are preset values, and C1-C4 are four different states in which the unmanned vehicle is located.

[0162] a vigilance value accumulation module, if a difference between a reward at the current time and a reward at a previous time exceeds a first threshold value, copying the environment at the current time to an environment replay pool and calculating a vigilance value accumulated to the current time, wherein the vigilance value is a vigilance value of the spiking neural network to the environment.

[0163] the first optimization module is configured to, if the vigilance value accumulated to the current time does not exceed a second threshold value, successively optimize the spiking neural network based on a spike-timing-dependent plasticity mechanism and a dopamine regulation mechanism; wherein the optimized spiking neural network is used to generate an obstacle avoidance action of the unmanned vehicle at a next time.

[0164] the second optimization module is configured to, if the vigilance value accumulated to the current time exceeds the second threshold value, optimize the spiking neural network based on an offline evolution method of a genetic algorithm according to the environment at the current time in the environment replay pool.

[0165] The device generates an obstacle avoidance action at the current moment according to a state sequence of the unmanned vehicle at the current moment and the cortex-basal ganglia-thalamus network, and optimizes the cortex-basal ganglia-thalamus network at the next moment based on the obstacle avoidance action at the current moment, in the optimization process of the cortex-basal ganglia-thalamus network, a vigilance value of the cortex-basal ganglia-thalamus network to the environment is accumulated according to a reward difference at adjacent moments, when the accumulated vigilance value does not exceed a second threshold, the cortex-basal ganglia-thalamus network is optimized based on a spike-timing-dependent plasticity mechanism and a dopamine regulation mechanism in turn, and when the accumulated vigilance value exceeds the second threshold, the cortex-basal ganglia-thalamus network is optimized based on an offline evolution method of a genetic algorithm, which can effectively improve the rapid adaptation and decision accuracy of the cortex-basal ganglia-thalamus network to a complex environment, so as to realize the obstacle avoidance method with rapid adaptation and accurate decision to different complex environments.

[0166] The present disclosure also relates to a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform the brain-like pulse reinforcement evolution unmanned obstacle avoidance method.

[0167] The present disclosure also relates to a computer device comprising one or more processors and one or more memories, one or more programs stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the brain-like pulse reinforcement evolution unmanned obstacle avoidance method.

[0168] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 one flow or multiple flows and / or blocks Figure 1means for performing the function specified by the block or blocks.

[0170] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 flow or flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 flow or flows and / or blocks Figure 1 steps of means for performing the function specified by the block or blocks.

[0172] The above merely provides an embodiment of the present application, but is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the claims of the present application.

Claims

1. A brain-like impulse-enhanced evolutionary method for obstacle avoidance in autonomous driving, characterized in that, include: Obtain the state sequence of autonomous vehicles in the lane scene at the current moment; Based on the state sequence and the spiking neural network, the obstacle avoidance action of the autonomous vehicle at the current moment is generated; where the spiking neural network is a cortical-basal ganglia-thalamic network. The obstacle avoidance action interacts with the current environment to generate a reward provided by the current environment; where the current environment is the environment in the lane scene at the current moment. If the difference between the reward at the current moment and the reward at the previous moment exceeds the first threshold, the current moment's environment is copied to the environment replay pool, and the accumulated alert value up to the current moment is calculated; where the alert value is the alert value of the spiking neural network to the environment; If the accumulated alert value up to the current moment does not exceed the second threshold, the spiking neural network is optimized sequentially based on the peak time-dependent plasticity mechanism and the dopamine regulation mechanism; the optimized spiking neural network is used to generate the obstacle avoidance action of the autonomous vehicle in the next moment. If the accumulated alert value up to the current moment exceeds the second threshold, the spiking neural network is optimized using an offline evolution method based on a genetic algorithm, according to the current environment in the environment replay pool.

2. The method according to claim 1, characterized in that, The spiking neuron A in the corticobasal ganglia-thalamus network was constructed based on the gene regulatory network A1 and the Izkovich neuron model A2. The Izkovich neuron model A2 was used to model the firing behavior of spiking neuron A; Gene regulatory network A1 is used to generate synaptic weights between spiking neuron A and spiking neuron B. The input of gene regulatory network A1 is the synaptic current of spiking neuron A; spiking neuron B is a spiking neuron that receives the output of spiking neuron A.

3. The method according to claim 1, characterized in that, The formula for generating the reward given by the environment at the current moment is: ； ； ； In the formula, r t s is the reward given by the environment at the current time t. t Let R be the state sequence of the autonomous vehicle at time t. b (s t ) for s t The corresponding total discrete reward, Eva(s) t ) for s t The corresponding continuous reward, α is a fixed coefficient, d safe d is the preset safe distance between autonomous vehicles and obstacles. current v represents the distance between the autonomous vehicle and the obstacle at the current moment. current R represents the speed of the driverless vehicle at the current moment. collision (s t R represents the discrete reward for an autonomous vehicle when it collides with an obstacle. speed (s t R is the discrete reward calculated based on the current speed interval of the autonomous vehicle. obstacle (s t R is the discrete reward for whether an autonomous vehicle maintains a safe distance from obstacles. lane-change (s t R represents the discrete reward for an autonomous vehicle successfully changing lanes to avoid an obstacle. fre-change (s t ) represents the discrete reward for continuous lane changes by the autonomous vehicle, y1 and y2 are preset values, and C1~C4 represent four different states of the autonomous vehicle.

4. The method according to claim 1, characterized in that, The formula for calculating the accumulated alert value up to the current moment is: ； In the formula, Let β be the accumulated alert value up to the current time t, let μ be the accumulated alert value up to the previous time, let μ be the update rate of δ(t), and let δ(t) be the difference between the reward at the current time and the reward at the previous time.

5. The method according to claim 1, characterized in that, The spiking neural network was optimized based on the spike-time-dependent plasticity mechanism and the dopamine regulation mechanism, including: The first optimization of the spiking neural network is based on the peak time-dependent plasticity mechanism. The first optimization is to optimize the synaptic weights between the dorsolateral prefrontal cortex and striatal D1 dopamine receptors, and between the dorsolateral prefrontal cortex and striatal D2 dopamine receptors in the spiking neural network. If the difference between the reward at the current moment and the reward at the previous moment is greater than 0, based on the first optimization, increase the synaptic weight between the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor, and decrease the synaptic weight between the dorsolateral prefrontal cortex and the striatal D2 dopamine receptor. If the difference between the reward at the current moment and the reward at the previous moment is not greater than 0, based on the first optimization, decrease the synaptic weight between the dorsolateral prefrontal cortex and the striatal D1 dopamine receptor, and increase the synaptic weight between the dorsolateral prefrontal cortex and the striatal D2 dopamine receptor.

6. The method according to claim 1, characterized in that, Based on the current environment in the environment replay pool, an offline evolutionary method based on a genetic algorithm is used to optimize the spiking neural network, including: S1) Obtain spiking neurons from the dorsolateral prefrontal cortex of the spiking neural network to be optimized, and use the obtained spiking neurons as the initial nuclei; S2) Randomly expand the initial kernel to obtain multiple new kernels. Replace the initial kernel in the spiking neural network to be optimized with the new kernels to obtain multiple new spiking neural networks to be optimized. S3) Traverse all spiking neural networks to be optimized, obtain the reward corresponding to the elite network and the reward corresponding to the spiking neural network to be optimized under the current environment, and take the difference between the reward corresponding to the initial elite network and the reward corresponding to the spiking neural network to be optimized as the reward of the corresponding kernel. Calculate the reward of each spiking neuron in the kernel based on the reward of the kernel and the synaptic weight of each spiking neuron in the kernel. Among them, the initial elite network is the spiking neural network with the highest reward at the historical time. S4) If the number of iterations reaches the third threshold, the top N spiking neurons with high rewards are used as the final nucleus to obtain the optimized spiking neural network. If the number of iterations does not reach the third threshold, the top N spiking neurons with high rewards are subjected to mutation and reproduction operations, and the spiking neurons after mutation and reproduction operations are used as the initial nucleus, and then proceed to S2). Wherein, N is the number of spiking neurons in the dorsolateral prefrontal cortex of the spiking neural network that has not been optimized using the offline evolution method.

7. A brain-like impulse-enhanced evolution-based unmanned obstacle avoidance device, characterized in that, include: The acquisition module acquires the state sequence of autonomous vehicles in the lane scene at the current moment; The obstacle avoidance action generation module generates the obstacle avoidance action of the autonomous vehicle at the current moment based on the state sequence and the spiking neural network; where the spiking neural network is a cortical-basal ganglia-thalamic network. The reward generation module interacts with the obstacle avoidance action and the current environment to generate a reward given by the current environment; where the current environment is the environment in the lane scene at the current moment. The alert value accumulation module, if the difference between the reward at the current moment and the reward at the previous moment exceeds the first threshold, copies the current moment's environment to the environment replay pool and calculates the alert value accumulated to the current moment; where the alert value is the alert value of the spiking neural network to the environment; The first optimization module optimizes the spiking neural network based on the spike time-dependent plasticity mechanism and the dopamine regulation mechanism if the accumulated alert value up to the current moment does not exceed the second threshold. The optimized spiking neural network is used to generate the obstacle avoidance action of the autonomous vehicle in the next moment. The second optimization module optimizes the spiking neural network based on an offline evolution method using a genetic algorithm, if the accumulated alert value up to the current moment exceeds the second threshold, according to the current moment's environment in the environment replay pool.

8. The apparatus according to claim 7, characterized in that, In the reward generation module, the reward given by the environment at the current moment is generated using the following formula: ； ； ； In the formula, r t s is the reward given by the environment at the current time t. t Let R be the state sequence of the autonomous vehicle at time t. b (s t ) for s t The corresponding total discrete reward, Eva(s) t ) for s t The corresponding continuous reward, α is a fixed coefficient, d safe d is the preset safe distance between autonomous vehicles and obstacles. current v represents the distance between the autonomous vehicle and the obstacle at the current moment. current R represents the speed of the driverless vehicle at the current moment. collision (s t R represents the discrete reward for an autonomous vehicle when it collides with an obstacle. speed (s t R is the discrete reward calculated based on the current speed interval of the autonomous vehicle. obstacle (s t R is the discrete reward for whether an autonomous vehicle maintains a safe distance from obstacles. lane-change (s t R represents the discrete reward for an autonomous vehicle successfully changing lanes to avoid an obstacle. fre-change (s t ) represents the discrete reward for continuous lane changes by the autonomous vehicle, y1 and y2 are preset values, and C1~C4 represent four different states of the autonomous vehicle.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 6.

10. A computer device, characterized in that, include: One or more processors and one or more memories, one or more programs stored in one or more memories and configured to be executed by one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 6.

Images