Event feature-based reinforcement learning decoupling training method

Abstract

The application provides an event feature-based reinforcement learning decoupling training method, which constructs a low-dimensional feature extraction network model and a high-dimensional feature extraction network model, extracts high-dimensional and low-dimensional features from event information received by an event camera, splices and fuses the extracted features, uses the fused features as state information of an interaction between a reinforcement learning intelligent agent and an environment, then inputs the state information into a policy network to obtain the action of a UAV, and gives reward feedback according to a self-defined reward function to complete the update of the policy network, and decouples the reinforcement learning training process into two stages, greatly optimizes the training efficiency and generalization ability of the reinforcement learning, and realizes the effective state information acquisition and obstacle avoidance of the UAV in a high-speed dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a reinforcement learning decoupling training method based on event characteristics. BACKGROUND

[0002] With the great progress of information technology and artificial intelligence, unmanned aerial vehicles, referred to as UAVs, are increasingly used in a wide range of applications, such as package delivery, target tracking, and map modeling, due to their high flexibility and maneuverability. UAVs can achieve different levels of autonomous flight or flight under remote control by humans. Typically, UAVs are equipped with cameras and other sensors to collect information about the surrounding environment, enabling the UAV to navigate autonomously in that environment. UAV navigation training is usually performed in a virtual simulation environment because the computational resources and power supply of UAVs are limited, and the cost of replacing UAV components due to crashes is expensive. In addition, the size and payload capacity of UAVs are also limited, so the on-board sensors often have to choose small, low-power sensors and minimize the computational load of the processing algorithms.

[0003] Drones often perform tasks in partially observable or unknown environments, in which it is difficult to define an accurate mathematical model. Deep Reinforcement Learning is a subfield of machine learning that uses concepts of Deep Learning (DL) and Reinforcement Learning (RL) to provide an optimal solution based on experience. This experience is based on iterations and evaluation of a reward function to determine the best behavior of an agent. Deep Reinforcement Learning has both powerful Deep Learning representation capabilities and excellent Reinforcement Learning decision-making capabilities, combining Artificial Neural Networks (ANN) with Reinforcement Learning by using a deep agent to learn an optimal policy. The type of Artificial Neural Network used in Deep Reinforcement Learning varies depending on the application, specifically depending on the problem being solved, the type of input (state), and the number of inputs passed to the Artificial Neural Network. For example, a Reinforcement Learning framework can be integrated with Convolutional Neural Networks (CNN) to process images representing the state of the environment, or combined with Recursive Neural Networks (RNN) to process inputs at different time steps. Obstacle avoidance, as a basic ability required for drones in any environment, can be learned and optimized through feedback reward values obtained through continuous interaction between the agent and the environment, using information obtained from different devices such as front-facing cameras or distance sensors as input for Deep Reinforcement Learning. The application of Deep Reinforcement Learning in the field of drones can help drones to learn autonomously, and advanced learning algorithms can help drones to traverse different obstacle environments with low risk.

[0004] Autonomous navigation is driven by tight coupling between perception and action, which is particularly challenging for agile, fast drones. Drones face the problem of obstacle avoidance, and low sensor latency is key to successfully performing agile maneuvers. Modern computer vision and machine learning techniques for perception and navigation typically focus on analyzing various forms of data from traditional cameras, such as RGB images, depth maps, etc. While these cameras provide high-resolution data, the main drawback of these sensors is their speed, with most averaging an output rate of 30-60 Hz. This makes this sensor unable to scale to the perception data rates required for agile navigation. Inspired by biological vision, neuromorphic engineering has given rise to a new sensor, called a dynamic vision sensor or event camera. Event cameras detect and measure changes in luminance on a per-pixel basis and return information about "events" with time resolution on the order of microseconds. Event cameras have high dynamic range, low latency, low power consumption, and many other advantages that traditional cameras do not have due to the increased sampling speed and small amount of data to be processed. This can allow the use of faster control schemes that quickly gather enough relevant environmental information to make intelligent control choices. In addition, due to the imaging principle of the event camera itself, it does not produce motion blur, effectively adapting to rapidly changing environments; and the event camera has the characteristics of high dynamic range, which can effectively image when the light intensity is large or small.

[0005] The basic working principle of the event-based camera is to measure the logarithmic luminance change at a given pixel position (x, y), specifically ΔlogI((x, y), t), where I is the luminosity intensity. When the change in logarithmic luminance exceeds a set threshold, the camera will generate an "event" reporting the time and location of the change, as well as the "sign" of the change. For an event camera with a resolution of (H, W), an event can be defined as a tuple e i = {x I , y I , p I , t i}, where x I and y I are the coordinates of the pixel, p I is the polarity of the event, t I is the timestamp, and e i refers to the i-th event in the event stream. The sequence of events over a time window τ can be represented as Eτ = {e i | t < i < t + τ}, where i indicates the index number. When a constant time window τ is slid over a longer event sequence, it can be seen that the length of Eτ will not be constant, as the number of events emitted within that interval will vary based on environmental or sensory considerations. The events in Eτ can also be accumulated and represented as a corresponding event image frame I Eτ .

[0006] Drones often operate in high-speed, dynamic environments. Traditional sensors on reinforcement learning agents typically address motion blur and other issues by increasing frame rates and the number of sensors. However, this leads to a significant waste of computing power, memory, and space due to the large amount of redundant information, making it difficult to meet the demands of increasingly complex applications. Event cameras, whose brightness changes are usually caused by motion, are well-suited for handling continuous state problems, making them ideal for learning control strategies. However, research combining event cameras with reinforcement learning methods is still in its early stages. Existing methods mostly combine traditional vision methods with reinforcement learning, processing event streams into frames and using traditional vision algorithms such as convolutional neural networks for feature extraction for reinforcement learning tasks. Furthermore, the visual representation of event cameras is fundamentally different; the data generated by event cameras is sparse and uneven. Effectively utilizing the data generated by event cameras is currently crucial for reinforcement learning applications. Summary of the Invention

[0007] To address the aforementioned issues, this invention equips UAVs with an event camera to process dynamic information in high-speed dynamic environments. This camera measures pixel brightness changes asynchronously with low latency, overcoming the limitations of traditional sensors. The invention proposes a feature fusion representation framework that combines event streams and event frames to process event information. Based on this framework, a two-stage decoupled reinforcement learning training method is proposed to help UAVs better evaluate and optimize their strategies, thereby enabling UAVs to effectively acquire state information and avoid obstacles in high-speed dynamic environments.

[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0009] A reinforcement learning decoupled training method based on event features includes the following steps:

[0010] Step S1: Extract the offline event stream and construct a training set. The training data includes offline event information and ground truth images, where the offline event information includes coordinate information, polarity, and timestamp.

[0011] Step S2: Construct a low-dimensional feature extraction network model based on an event-based variational autoencoder and normalize the timestamps of the event sequence.

[0012] Step S3: Obtain event frames based on the logarithmic intensity difference of image pixels at a time threshold, and construct a high-dimensional feature extraction network model;

[0013] Step S4: Initialize the pre-trained model, i.e. the low-dimensional feature extraction network model. The low-dimensional feature extraction network model is pre-trained using offline event stream information. This model consists of event image reconstruction loss and KL divergence loss.

[0014] Step S5: Initialize the reinforcement learning network and the high-dimensional feature extraction network model, configure the training hyperparameters, and train these two network models;

[0015] Step S6: Input the event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, and then perform feature splicing and fusion on the obtained low-dimensional and high-dimensional information. The fused information is used as the state information of the reinforcement learning agent's interaction with the environment.

[0016] Step S7: Construction of the UAV motion space, specifically as follows:

[0017]

[0018] θ = radians(a)

[0019] Where z represents the final output value after feature fusion, π is the policy network of the reinforcement learning algorithm, λ is the hyperparameter weight, the action space is first mapped to the interval [-1,1], and θ is the drone flight radian, which is obtained by radianizing action a.

[0020] Step S8: Construct a policy network and set the corresponding reward function. The policy network is used to obtain the motion policy through reinforcement learning.

[0021] Step S9: Set up a virtual simulation environment, construct training and testing scenarios, and use the virtual simulation environment for reinforcement learning training. Configure parameters including the total number of training steps, learning rate, batch size, number of training rounds, and flight step size. During training, the input event stream information will be directly input into the low-dimensional feature extraction network to obtain the latent vector Z. s After aggregating event stream information into event frames, the data is input into a high-dimensional feature extraction network to obtain the latent vector Z. f , will Z s and Z f After being spliced ​​and fused, it becomes the state information S. Then, the motion policy is obtained from the action space based on the policy network using the state information.

[0022] Furthermore, in step S2, the timestamp is normalized using a composite phase function with composite rotation, and the time encoding formula modifies the features as follows:

[0023]

[0024] Among them, z i Indicates the input event, Δt j，i τ represents the time difference between events i and j, and τ represents the time window.

[0025] To better extract temporal feature information from the event stream, we first separate the temporal (t) and spatial (x, y, p) characteristics of each event e in the event stream, where x and y are the pixel coordinates and p is the event polarity. Then, we use temporal embedding to normalize the timestamps to the range [0, 1], where the timestamp corresponding to the end of the window is mapped to 1. The d-dimensional temporal feature of each normalized timestamp is calculated as follows:

[0026]

[0027] Where i represents the event number. The formula encodes the standardized timestamp as a combination of sine / cosine values ​​and then embeds it into the event features.

[0028] Further, in step S3, an event frame is obtained based on the logarithmic intensity difference of pixels at a time threshold, specifically as follows:

[0029] By comparing the differences between two RGB images, the logarithmic intensity difference of each pixel at a time threshold is calculated, and the result is compared with the threshold. The difference in logarithmic intensity between the current frame and the previous frame is taken, specifically:

[0030] ΔL(u,t)=log(I t )-log(I t-1 )

[0031] The polarity of each pixel is calculated based on the change threshold of the logarithmic intensity (THR), specifically as follows:

[0032]

[0033] Based on the intensity change exceeding the threshold, determine the number of events to be triggered for each pixel. Let Nmax be the maximum number of events that can occur at a single pixel, specifically:

[0034]

[0035] This determines the timestamps between events, as follows:

[0036]

[0037] Furthermore, in step S4, a low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data. This model consists of event image reconstruction loss and KL divergence loss, as detailed below:

[0038]

[0039] The first term represents the reconstruction loss, P(z) is the distribution of the latent variable, and Q(z|x) is the approximate posterior of the variational autoencoder. Further, in step S8, the reward function for the UAV is set as follows:

[0040] r t =r g，t +r c，t +r k，t +r p，t

[0041]

[0042] r k，t =k d (||p t-1 -g||-||p t -g||)

[0043] r p，t =-(λ·Penalty(t))

[0044] Where, r g，t To reach the reward, r c，t For collision penalty, r k，t For process rewards, r p，t For step penalty, p t p represents the current location of the drone. t-1 This indicates the drone's position at the previous time step, g represents its destination, collison_state contains collision information, and r... collision For the collision score, k d Representing distance, Penalty(t) is a linear penalty that increases with the number of moves in a round, and λ is a hyperparameter that controls the weight of the penalty term.

[0045] Furthermore, in step S8, the reinforcement learning policy network used is the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm has two variants. The first variant is penalty-based, subtracting the KL divergence bias between the old and new policies from the objective function and multiplying by a constant C. This allows for the use of a simple stochastic gradient descent algorithm to optimize the function, as detailed below:

[0046]

[0047] Where a represents the UAV's action space, s represents the state information, θ represents the policy parameters, J(θ) represents the objective function, and π θ Represents the policy network, π θ (a|s) represents the policy network π θ The action taken in state s is a, and A represents the dominance function;

[0048] The second variant is based on pruning, which relies on pruning the objective function to ensure that the parameters do not change too much, as follows:

[0049]

[0050] Where ∈ is a hyperparameter, and clip represents the clipping function, used to limit the degree of update between the old and new strategies. This indicates that action a has a higher value than average. It will increase, but will not exceed 1 + ∈; if It will decrease, but will not exceed 1-∈.

[0051] The beneficial technical effects of the present invention are as follows:

[0052] This invention provides a decoupled training method for reinforcement learning based on event features. The scheme applies low-dimensional and high-dimensional feature representation models and reinforcement learning to UAV obstacle avoidance. To address the asynchronous and uneven nature of event data, low-dimensional information from the event stream and high-dimensional information from event frames are extracted separately. Low-dimensional and high-dimensional feature extraction network models are constructed, and the fused features are used as environmental state information in reinforcement learning. This state information is then input into the policy network to obtain the UAV's actions, and a reward feedback is provided according to a custom reward function to update the policy network. Based on this, a two-stage decoupled reinforcement learning training method is proposed. The low-dimensional feature extraction network model is pre-trained, and the reinforcement learning network and high-dimensional feature extraction network models are iteratively trained, greatly optimizing the training efficiency and generalization ability of reinforcement learning. Through this scheme, this invention demonstrates excellent performance in extracting environmental information from UAVs in high-speed dynamic environments, increasing the success rate of UAVs in high-speed obstacle avoidance tasks, and enabling UAVs to effectively acquire state information and avoid obstacles in high-speed dynamic environments. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a reinforcement learning decoupling training method based on event features provided in an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the virtual simulation environment provided in an embodiment of the present invention;

[0056] Figure 3 This is a flowchart of the drone testing process provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and the following embodiments.

[0059] like Figure 1 As shown, a reinforcement learning decoupling training method based on event features includes the following steps:

[0060] Step S1: Extract the offline event stream and construct a training set. The training data includes offline event information and ground truth images, where the offline event information includes coordinate information, polarity, and timestamp.

[0061] Step S2: Construct a low-dimensional feature extraction network model based on an event-based variational autoencoder and normalize the timestamps of the event sequence.

[0062] Step S3: Obtain event frames based on the logarithmic intensity difference of image pixels at a time threshold, and construct a high-dimensional feature extraction network model;

[0063] Step S4: Initialize the pre-trained model, i.e. the low-dimensional feature extraction network model. The low-dimensional feature extraction network model is pre-trained using offline event stream information. This model consists of event image reconstruction loss and KL divergence loss.

[0064] Step S5: Initialize the reinforcement learning network and the high-dimensional feature extraction network model, configure the training hyperparameters, and train these two network models;

[0065] Step S6: Input the event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, and then perform feature splicing and fusion on the obtained low-dimensional and high-dimensional information. The fused information is used as the state information of the reinforcement learning agent's interaction with the environment.

[0066] Step S7: Construction of the UAV motion space, specifically as follows:

[0067]

[0068] θ = radians(a)

[0069] Where z represents the final output value after feature fusion, π is the policy network of the reinforcement learning algorithm, λ is the hyperparameter weight, the action space is first mapped to the interval [-1,1], and θ is the drone flight radian, which is obtained by radianizing action a.

[0070] Step S8: Construct a policy network and set the corresponding reward function. The policy network is used to obtain the motion policy through reinforcement learning.

[0071] Step S9: Set up a virtual simulation environment, construct training and testing scenarios, and use the virtual simulation environment for reinforcement learning training. Configure parameters including the total number of training steps, learning rate, batch size, number of training rounds, and flight step size. During training, the input event stream information will be directly input into the low-dimensional feature extraction network to obtain the latent vector Z. s After aggregating event stream information into event frames, the data is input into a high-dimensional feature extraction network to obtain the latent vector Z. f , will Z s and Z f After being spliced ​​and fused, it serves as the state information S. During this period, the parameters in the low-dimensional feature extraction network remain unchanged, while the high-dimensional feature extraction network is updated as training progresses. Then, the motion policy is obtained from the action space using the state information and the policy network.

[0072] In step S2, using timestamps as part of the main feature calculation can interfere with recursive processing. Therefore, effectively utilizing timestamp information is crucial. This invention normalizes timestamps using a composite phase function with composite rotation. The modification of the features by the time encoding formula is described as follows:

[0073]

[0074] Among them, z i Indicates the input event, Δt j，i τ represents the time difference between events i and j, and τ represents the time window.

[0075] Directly using time data as input to a preprocessing neural network interferes with feature calculation because global timestamps are arbitrary values, and even the relative time difference between each event changes with each new event received, necessitating feature recalculation. To better extract time feature information from the event stream, we first separate the time t and spatial (x, y, p) characteristics of each event e in the event stream. Then, we use time embedding to normalize the timestamps to the range [0, 1], where the timestamp corresponding to the end of the window is mapped to 1. The calculation of the d-dimensional time feature for each normalized timestamp is described below:

[0076]

[0077] in, The formula encodes a standardized timestamp as a combination of sine / cosine values, then embeds it into an event feature, and subsequently decodes it into a value indicating the recentity of the event.

[0078] In step S3, an event frame is obtained based on the logarithmic intensity difference of pixels at a time threshold, specifically:

[0079] By comparing the differences between two RGB images, the logarithmic intensity difference of each pixel at a time threshold is calculated, and the result is compared with the threshold. The difference in logarithmic intensity between the current frame and the previous frame is taken, as described in detail below:

[0080] ΔL(u,t)=log(I t )-log(I t-1 )

[0081] The polarity of each pixel is calculated based on the threshold of change in logarithmic intensity (THR), as described below:

[0082]

[0083] Based on the intensity change exceeding a threshold, the number of events to be triggered for each pixel is determined. Let N_max be the maximum number of events that can occur at a single pixel, as described below:

[0084]

[0085] This determines the timestamps between events, as described below:

[0086]

[0087] In step S4, a low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data. This model consists of event image reconstruction loss and KL divergence loss, as described in detail below:

[0088]

[0089] The first term represents the reconstruction loss, P(z) is the distribution of the latent variable, and Q(z|x) is the approximate posterior of the variational autoencoder.

[0090] During training, batched data is used as input to the event-based variational autoencoder. Specifically, 50 batches are set, each containing 2000 events. Batch selection is achieved by randomly sampling indices from the entire event sequence and then reading the number of subsequent events equal to the selected batch size. An Adam optimizer with a learning rate of 0.001 is used, and the training proceeds for 20,000 iterations. Furthermore, to avoid the vanishing KL divergence problem, an annealing technique is introduced. A varying weight is applied to the KL loss over the following scheduling period, which is also appropriately weighted based on the value of the reconstruction loss, as described below: Where β is the weight;

[0091] In step S8, the reward function for the UAV is designed, including the following parts:

[0092] r t =r g，t +r c，t +r k，t +r p，t

[0093] When the drone reaches its destination, it is given a size of r. arrival The reward, r g，t To reach the reward, it is represented as follows, where p t This indicates the location of the drone, and g represents its destination:

[0094]

[0095] When a drone collides, the virtual simulation environment returns collision information `collison_state` based on the collision conditions. When the returned state is true, the drone will receive a collision score `r`. collision The punishment, r c，t Collision penalty:

[0096]

[0097] Introducing process rewards to provide more feedback information guides the agent to find effective behavioral strategies more quickly, thereby accelerating the learning process and overcoming the difficulties caused by the sparse reward problem. k，t For process rewards, the following is represented:

[0098] r k，t =k d (||p t-1 -g||-||p t -g||)

[0099] Where, k d p represents distance. t p represents the current location of the drone.t-1 This indicates the drone's previous time step position. As the drone gets closer to its destination, it can still receive process rewards, encouraging the drone to actively explore in the direction of progress.

[0100] By employing reward decay or reward weighting mechanisms to adjust the reward weights in different states, the agent can focus more on key task points or important decisions, accelerating the learning convergence and optimization process. p，t The step count penalty is described in detail below:

[0101] r p，t =-(λ·Penalty(t))

[0102] Penalty(t) is a linear penalty, meaning the more steps there are in a round, the greater the penalty. λ is a hyperparameter that controls the weight of the penalty term.

[0103] In step S8, the reinforcement learning policy network used is the Proximal Policy Optimization (PPO) algorithm, and the empirical update method of PPO is as follows:

[0104]

[0105] Where θ represents the policy parameters, and J(θ) represents the objective function, which optimizes the parameters in the objective function using θ; π θ Let a represent a policy network with parameter θ. t and s t Let π represent the actions and states of the UAV at time t, respectively. θ (a t |s t ) represents the policy network π θ In s t The action taken in the state is a t R(t) represents the reward value at time t;

[0106] There are two main variants of the PPO algorithm. The first variant is penalty-based, aiming to address the overhead problem of the Trust Region Policy Optimization (TRPO) algorithm by incorporating constraints as penalties into the objective function. Instead of adding separate constraints, this variant subtracts the KL divergence deviation between the old and new policies from the objective function and multiplies it by a constant C. This allows for optimization using simple stochastic gradient descent, as described below:

[0107]

[0108] Where a represents the drone's action space, s represents state information, and π θLet C represent the policy network and A represent the advantage function. The difficulty of the first variant lies in choosing a suitable value for C. The algorithm updates C based on the magnitude of the KL divergence. If the KL divergence is too high, C increases; if the KL divergence is too low, C decreases. The second variant is based on pruning. It does not use KL divergence in the objective function and has no constraints whatsoever. Instead, it relies on specifically pruning the objective function to ensure that the parameter changes are not too large. The details are as follows:

[0109]

[0110] Where ∈ is a hyperparameter, and clip represents the clipping function, used to limit the degree of update between the old and new strategies. This indicates that action a has a higher value than average. It will increase, but will not exceed 1 + ∈. Similarly, if It will decrease, but will not exceed 1-∈.

[0111] In step S9, four obstacle avoidance scenarios were created in the AirSim virtual simulation environment, such as... Figure 2 As shown, these scenarios are named Test, Testlong, ChangeShape, and Train. The Train scenario offers the most straightforward layout of the four scenarios. Compared to the others, it is narrower, has fewer planks, and the training environment is approximately 100 meters long with a wall on each side, making the overall width about 30 meters. The coordinate axis is centered at the bottom of the environment's central axis, with the lateral coordinate X, the longitudinal forward coordinate Y, and the altitude coordinate Z. The initial starting point X coordinate is randomly within the range [-10, 10], Y is set to 0, and Z is set to 5. This scenario is specifically designed for training, providing more positive feedback for reinforcement learning strategies and helping the agent train more effectively by reducing scenario complexity. The Testlong scenario maintains the same width as the Train scenario but significantly increases the density of planks. This scenario aims to test the drone's ability to navigate in a dense obstacle field. Finally, the ChangeShape scenario has a similar layout to the Test scenario but uses ellipsoids instead of planks. This presents a significant challenge for drone models trained only in scenarios characterized by planks, as they must adapt to the new shape and features of the ellipsoid. In each scenario, the drone takes off from a designated starting point, with the objective of reaching a destination (100 meters) while successfully avoiding dense obstacles. If the drone collides with an obstacle, the mission is considered a failure, and the drone is reset to the initial starting point. In the Testlong scenario, the initial point is set at the center of the starting point, while in the other three scenarios, the starting point is randomly determined within the initial area. Furthermore, it is assumed that the drone moves at a constant, predefined speed, and the step size of its actions varies according to the desired frequency.

[0112] This embodiment tested the above three test scenarios, and the test methods are as follows: Figure 3 As shown, firstly, the pre-trained fusion event feature network model is loaded, including a low-dimensional feature extraction network model, a high-dimensional feature extraction network model, and a reinforcement learning policy network. Then, a simulation test environment is constructed, setting the drone's action space, starting and ending points, obstacle types, obstacle sizes, etc. Finally, obstacle avoidance tests are performed on the drone, initializing test metrics and configuring test parameters. In each case, the drone starts flying from the initial starting point and resets upon successfully avoiding an obstacle and reaching the destination, or upon colliding with an obstacle. A total of 1000 obstacle avoidance tests were conducted, and the number of times the destination was successfully reached was recorded, as shown in the table below:

[0113]

[0114] It is evident that the solution of this invention has excellent performance in extracting environmental information in high-speed dynamic environments, which can increase the success rate of UAVs in obstacle avoidance tasks during high-speed movement.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event feature-based reinforcement learning decoupling training method, characterized in that, The method comprises: Step S1: extracting an offline event stream, constructing a training set, and training data including offline event information and true value images, wherein the offline event information includes coordinate information, polarity and time stamp; Step S2: based on the event-based variational autoencoder, and normalizing the time stamp of the event sequence, a low-dimensional feature extraction network model is constructed; Step S3: based on the log intensity difference of the image pixels within a time threshold, an event frame is obtained, and a high-dimensional feature extraction network model is constructed; Step S4: initializing a pre-trained model, i.e. a low-dimensional feature extraction network model, pre-training the low-dimensional feature extraction network model using offline event stream information, the model consisting of event image reconstruction loss and KL divergence loss; Step S5: initializing a reinforcement learning network and a high-dimensional feature extraction network model, configuring training hyperparameters, and training the two network models; Step S6: inputting event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, then performing feature splicing and fusion on the obtained low-dimensional and high-dimensional information, and using the fused information as state information for interaction between the reinforcement learning agent and the environment; Step S7: constructing a UAV action space, which is specifically represented as follows: theta = radians(a) Wherein z represents the final output value after feature fusion, pi is the policy network of the reinforcement learning algorithm, lambda is the weight of the hyperparameter, the action space is first mapped to the interval [-1, 1], and theta is the radian of the UAV flight, obtained by radianizing the action a; Step S8: constructing a policy network and setting a corresponding reward function, the policy network being used to obtain a motion strategy through reinforcement learning; Step S9: a virtual simulation environment is built, a training and testing scene is constructed, reinforcement learning training is performed by using the virtual simulation environment, and parameters including a total number of model training steps, a learning rate, a batch size, a training round number, and a flight step length are configured. During the training process, the input event stream information is directly input into the low-dimensional feature extraction network to obtain the latent vector Z s After the event stream information is aggregated into an event frame, the high-dimensional feature extraction network is input to obtain the latent vector Z f Z s and Z f are spliced and fused to serve as state information S, and then the state information is used to obtain a motion strategy from an action space according to a policy network.

2. The method of claim 1, wherein, The step S2 further comprises: The time stamp is normalized by a composite phase function of composite rotation, and the modification of the time encoding formula to the feature is as follows: where z i represents an input event, Δt j，i represents the time difference between event i and event j, and τ represents a time window; In order to better extract time feature information from the event stream, first separate the time t and spatial (x, y, p) characteristics of each event e in the event stream, x and y are the coordinates of the pixels, and p is the polarity of the event, then use time embedding to normalize the time stamp to the range [0, 1], wherein the time stamp corresponding to the end of the window is mapped to 1, and the calculation of the d-dimensional time feature of each normalized time stamp is as follows: where i denotes the event sequence number, t e [0, 1], which encodes the normalized timestamp as a combination of sine / cosine values, which are then embedded into the event features.

3. The method of claim 1, wherein, In step S3, the event frame is obtained based on the log intensity difference of the pixels within a time threshold, including: By comparing the difference between two frames of RGB images, the log intensity difference of each pixel within a time threshold is calculated, and the result is compared with a threshold, taking the difference between the log intensity of the current frame and the previous frame, which is specifically: ΔL(u, t) = log(I t ) - log(I t-1 ) Based on the change threshold of the log intensity THR, the polarity of each pixel is calculated, which is specifically: According to the intensity change degree exceeding the threshold, the number of events to be triggered by each pixel is determined, and Nmax is the maximum number of events that can occur at a single pixel, which is specifically: The timestamps between events are thus determined, in particular:

4. The method of claim 1, wherein, In step S4, the low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data, and the model consists of event image reconstruction loss and KL divergence loss, which is specifically as follows: Wherein the first term represents the reconstruction loss, P(z) is the distribution of the latent variable, and Q(z|x) is the approximate posterior of the variational autoencoder.

5. The method of claim 1, wherein, In step S8, the reward function of the UAV is set as: r t ＝r g,t +r c，t +r k，t +r p，t r k,t = k d (||p t-1 -g||-||p t -g||) r p,t = -(λ · Penalty(t)) where r g，t is the reward to reach the goal, r c，t is the penalty for collision, r k,t is the process reward, r p，t is the step penalty, p t denotes the current position of the UAV, p t-1 denotes the position of the UAV at the previous time step, g denotes its goal, and collision_state is the collision information, r collision is the collision score, k d denotes the distance, and Penalty(t) is a linear penalty that increases with the number of steps in the episode, with λ being a hyperparameter that controls the weight of the penalty term.

6. The method of claim 1, wherein, In step S8, the reinforcement learning policy network adopted is the proximal policy optimization (PPO) algorithm, which has two variants. The first variant is based on penalty, subtracting the KL divergence deviation between the new and old policies from the target function and multiplying it by a constant C, allowing the function to be optimized using a simple stochastic gradient descent algorithm, as follows: Wherein, a represents the action space of the unmanned aerial vehicle, s represents the state information, θ represents the policy parameter, J(θ) represents the target function, π θ represents the policy network, π θ (a|s) represents the policy network π θ The action taken in the s state is a, and A represents the advantage function; The second variant is based on clipping, which relies on clipping the target function to ensure that the change in parameters is not too large, as follows: where ∈ is a hyperparameter, and clip denotes a clipping function that limits the extent of the update between the old and new policies, if the value of action a is above average, will increase, but not by more than 1+∈; if will decrease, but not by more than 1-∈.

Images