Reinforcement learning decoupling training method based on event features

Abstract

The invention provides a reinforcement learning decoupling training method based on event features, and the method comprises the steps: constructing a low-dimensional feature extraction network model and a high-dimensional feature extraction network model, carrying out the extraction of high-dimensional and low-dimensional features of event information received by an event camera, carrying out the splicing and fusion of the extracted features, and carrying out the recognition of the high-dimensional and low-dimensional features. The fused features serve as state information of interaction between a reinforcement learning agent and the environment, then the state information is input into a strategy network to obtain actions of the unmanned aerial vehicle, reward feedback is given according to a self-defined reward function to complete updating of the strategy network, and the reinforcement learning training process is decoupled into two stages; the training efficiency and generalization ability of reinforcement learning are greatly optimized, and effective state information acquisition and obstacle avoidance of the unmanned aerial vehicle in a high-speed dynamic environment are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a reinforcement learning decoupling training method based on event features. Background Art

[0002] With the great progress of information technology and artificial intelligence, unmanned aerial vehicles (UAVs) are increasingly being used in a wide range of applications, such as package delivery, target tracking, and map modeling, due to their high flexibility and high maneuverability. UAVs can achieve varying degrees of autonomous flight or fly under remote control under human operation. Typically, UAVs are equipped with cameras and other sensors to collect information about the surrounding environment, enabling the UAV to navigate autonomously in that environment. UAV navigation training is usually conducted in a virtual simulation environment because UAVs have limited computing resources and power supply, and it is expensive to replace UAV parts due to crashes. In addition, the size and payload capacity of UAVs are also limited, so onboard sensors can often only choose small, low-power sensors and minimize the computational load of processing algorithms.

[0003] Drones often perform tasks in partially observable or unknown environments where it is difficult to define precise mathematical models. Deep reinforcement learning is a subfield of machine learning that uses deep learning (DL) concepts and reinforcement learning (RL) to provide optimal solutions based on experience. This experience is based on iterating and evaluating reward functions to determine the best behavior of the agent. Deep reinforcement learning not only has powerful deep learning representation capabilities, but also has excellent reinforcement learning decision-making capabilities. It combines artificial neural networks (ANN) with reinforcement learning by using deep agents to learn optimal policies. The types of artificial neural networks used in deep reinforcement learning vary from application to application, depending on the problem being solved, the type of input (state), and the number of inputs passed to the artificial neural network. For example, the reinforcement learning framework can be integrated with convolutional neural networks (CNN) to process images representing the state of the environment, or combined with recursive neural networks (RNN) to process inputs of different time steps. Obstacle avoidance is a basic capability required by drones in any environment. It can be learned and optimized by using information obtained from different devices (such as front cameras or distance sensors) as input for deep reinforcement learning, and by using the feedback reward values ​​obtained during the continuous interaction between the agent and the environment. Applying deep reinforcement learning to the drone field can help drones learn autonomously, and advanced learning algorithms can help drones traverse different obstacle environments with low risk.

[0004] Autonomous navigation is driven by the tight coupling between perception and action, and is particularly challenging for agile and fast unmanned aerial vehicles (UAVs). Facing the obstacle avoidance problem, low sensor latency is the key to successfully performing agile maneuvers. Modern computer vision and machine learning technologies for perception and navigation typically focus on analyzing various forms of data from traditional cameras, such as RGB images, depth maps, etc. Although these cameras provide high-resolution data, the main drawback of these sensors is their speed, with most having an average output rate of 30 - 60Hz. This makes such sensors unable to scale to the perception data rate required for agile navigation. Inspired by biological vision, neuromorphic engineering has produced a new type of sensor called a dynamic vision sensor or event camera. Event cameras detect and measure changes in luminance on a per-pixel basis and return information about "events" with a time resolution on the order of microseconds. Due to the increased sampling speed and small amount of data processing to be parsed, event cameras have many advantages such as high dynamic range, low latency, and low power consumption that traditional cameras do not have. This allows for the use of faster control schemes to quickly collect sufficient relevant environmental information for making informed control choices. In addition, due to the imaging principle of event cameras themselves, motion blur is not generated, effectively adapting to rapidly changing environments; and event cameras have the characteristic of high dynamic range, being able to effectively image both when the light intensity is large or small.

[0005] Given a pixel position (x, y), the basic working principle of an event-based camera is to measure the change in logarithmic luminance at that pixel, specifically ΔlogI((x, y), t), where I is the photometric intensity. When the change in logarithmic luminance exceeds a set threshold, the camera generates an "event" reporting the time and location of the change, as well as the "sign" of the change. For an event camera with a resolution of (H, W), an event can be defined as a tuple e of the following four quantities i ={x I , y I , p I , t i}, where x I and y I are the coordinates of the pixel, p I is the polarity of the event, t I is the timestamp, and e i refers to the i-th event in the event stream. The sequence of events over a τ time window can be represented as Eτ = {e i | t < i < t + τ}, where i represents the sequence number. When sliding a constant time window of τ over a longer event sequence, it can be seen that the length of Eτ will not be constant because the number of events emitted within that interval will vary based on environmental or sensory considerations. The events in Eτ can also be accumulated and represented as a corresponding event image frame I Eτ .

[0006] Drones often operate in high-speed dynamic environments. Traditional sensors carried by reinforcement learning agents often solve problems such as motion blur by increasing frame rates and the number of sensors. A large amount of redundant information will greatly waste computing power, memory and space. This approach has been unable to meet the needs of increasingly complex application scenarios. Since brightness changes in event cameras are usually caused by motion, they are conducive to dealing with continuous state problems, which makes event cameras very suitable for learning control strategies. However, the research on event cameras combined with reinforcement learning methods is still in its early stages. Most existing methods combine traditional visual methods with reinforcement learning, process event streams into frames, and use traditional visual algorithms such as convolutional neural networks for feature extraction for reinforcement learning tasks. In addition, the visual representation of event cameras is fundamentally different. The data generated by event cameras has sparse and uneven characteristics. How to effectively use the data generated by event cameras is the key to current reinforcement learning applications. Summary of the invention

[0007] In order to solve the above problems, the present invention equips drones with event cameras to process dynamic information in high-speed dynamic environments, measures pixel brightness changes asynchronously with low latency, breaks through the limitations of traditional sensors, and proposes a feature fusion representation framework that combines event streams and event frames to process event information. Based on the framework, a two-stage decoupled reinforcement learning training method is proposed to help drones better evaluate and optimize strategies, so as to enable drones to effectively obtain state information and avoid obstacles in high-speed dynamic environments.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A reinforcement learning decoupling training method based on event features comprises the following steps:

[0010] Step S1: extract offline event streams and construct training sets, where the training data includes offline event information and true value images, where the offline event information includes coordinate information, polarity, and timestamp;

[0011] Step S2: Based on the event-based variational autoencoder, the timestamps of the event sequence are normalized to build a low-dimensional feature extraction network model;

[0012] Step S3: obtaining an event frame based on the logarithmic intensity difference of image pixels at a time threshold, and constructing a high-dimensional feature extraction network model;

[0013] Step S4: Initialize a pre-trained model, i.e., a low-dimensional feature extraction network model, and pre-train the low-dimensional feature extraction network model using offline event stream information. The model consists of an event image reconstruction loss and a KL divergence loss.

[0014] Step S5: Initialize the reinforcement learning network and the high-dimensional feature extraction network model, configure training hyperparameters, and train the two network models;

[0015] Step S6: input the event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, and then perform feature splicing and fusion on the obtained low-dimensional and high-dimensional information, and use the fusion as the state information of the interaction between the reinforcement learning agent and the environment;

[0016] Step S7: UAV action space construction, specifically expressed as follows:

[0017]

[0018] θ=radians(a)

[0019] Among them, z represents the final output value after feature fusion, π is the policy network of the reinforcement learning algorithm, λ is the hyperparameter weight, the action space is first mapped to the [-1,1] interval, θ is the flight arc of the drone, which is obtained by radianizing the action a;

[0020] Step S8: construct a policy network and set a corresponding reward function. The policy network is used to obtain a motion strategy through reinforcement learning.

[0021] Step S9: Build a virtual simulation environment, construct training and test scenarios, and use the virtual simulation environment for reinforcement learning training. Configure the total number of steps for model training, learning rate, batch size, number of training rounds, and flight step length parameters. During the training process, the input event stream information will be directly input into the low-dimensional feature extraction network to obtain the potential vector Z s After aggregating the event stream information into event frames, it is input into the high-dimensional feature extraction network to obtain the latent vector Z f , Z s and Z f After splicing and fusion, it is used as the state information S, and then the state information is used to obtain the motion strategy from the action space according to the strategy network.

[0022] Furthermore, in step S2, the timestamp is normalized by a composite phase function of a composite rotation, and the time encoding formula modifies the feature as follows:

[0023]

[0024] Among them, z i represents the input event, Δt j，i represents the time difference between event i and event j, and τ represents the time window.

[0025] In order to better extract temporal feature information from the event stream, we first separate the temporal t and spatial (x, y, p) characteristics of each event e in the event stream, where x and y are the coordinates of the pixel and p is the polarity of the event. Then, we use time embedding to normalize the timestamp to the range [0, 1], where the timestamp corresponding to the end of the window is mapped to 1. The d-dimensional temporal feature of each normalized timestamp is calculated as follows:

[0026]

[0027] Where i represents the event number, This formula encodes the normalized timestamp as a combination of sine / cosine values, which are then embedded into the event features.

[0028] Further, in step S3, the event frame is obtained based on the logarithmic intensity difference of the pixel at the time threshold, specifically:

[0029] By comparing the difference between the two frames of RGB images, the logarithmic intensity difference of each pixel at the time threshold is calculated, and the result is compared with the threshold, and the difference in logarithmic intensity between the current frame and the previous frame is taken, specifically:

[0030] ΔL(u, t) = log(I t )-log(I t-1 )

[0031] The polarity of each pixel is calculated based on the threshold of the change in logarithmic intensity THR, specifically:

[0032]

[0033] According to the intensity change degree exceeding the threshold, the number of events to be excited for each pixel is determined. Let Nmax be the maximum number of events that can occur at a single pixel, specifically:

[0034]

[0035] This determines the timestamps between events as follows:

[0036]

[0037] Further, in step S4, a low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data, which consists of event image reconstruction loss and KL divergence loss, as follows:

[0038]

[0039] The first term represents the reconstruction loss, P(z) is the distribution of the latent variable, and Q(z|x) is the approximate posterior of the variational autoencoder. Further, in step S8, the reward function of the drone is set to:

[0040] r t =r g，t +r c，t +r k，t +r p，t

[0041]

[0042] r k，t =k d (||p t-1 -g||-||p t -g||)

[0043] r p，t = -(λ·Penalty(t))

[0044] Among them, r g，t To reach the reward, r c，t is the collision penalty, r k，t is the process reward, r p，t is the step penalty, p t Indicates the current position of the drone, p t-1 represents the position of the drone in the previous time step, g represents its destination, collison_state is the collision information, r collision is the collision score, k d represents the distance, Penalty(t) is a linear penalty. The more steps there are in a round, the greater the penalty. λ is a hyperparameter that controls the weight of the penalty term.

[0045] Further, in step S8, the reinforcement learning policy network adopted is the proximal policy optimization PPO algorithm. There are two variants of the PPO algorithm. The first variant is based on penalty. The KL divergence deviation between the new and old policies is subtracted from the objective function and multiplied by a constant C, allowing the use of a simple stochastic gradient descent algorithm to optimize the function, as follows:

[0046]

[0047] Among them, a represents the action space of the drone, s represents the state information, θ represents the strategy parameter, J(θ) represents the objective function, and π θ represents the policy network, π θ (a|s) represents the policy network π θ The action taken in state s is a, and A represents the advantage function;

[0048] The second variant is based on clipping, which relies on clipping the objective function to ensure that the changes in parameters are not too large, as follows:

[0049]

[0050] Where ∈ is a hyperparameter, and clip represents the clipping function, which is used to limit the degree of update between the new and old strategies. If This means that the value of action a is higher than the average. will increase, but will not exceed 1+∈; if will decrease, but will not exceed 1-∈.

[0051] The beneficial technical effects of the present invention are as follows:

[0052] The present invention provides a reinforcement learning decoupling training method based on event features. The scheme applies low-dimensional and high-dimensional feature representation models and reinforcement learning to unmanned aerial vehicle obstacle avoidance. In order to solve the asynchronous and uneven characteristics of event data, low-dimensional information of event streams and high-dimensional information in event frames are extracted respectively, and low-dimensional feature extraction network models and high-dimensional feature extraction network models are constructed. The features after the fusion of low-dimensional and high-dimensional extracted information are used as environmental state information in reinforcement learning. Subsequently, the state information is input into the policy network to obtain the action of the unmanned aerial vehicle, and reward feedback is given according to a custom reward function to complete the update of the policy network. On this basis, a reinforcement learning training method based on two-stage decoupling is proposed, and the low-dimensional feature extraction network model is pre-trained respectively, and the reinforcement learning network and the high-dimensional feature extraction network model are iteratively trained, which greatly optimizes the training efficiency and generalization ability of reinforcement learning. Through the above scheme, the present invention has good performance in extracting environmental information in the unmanned aerial vehicle facing a high-speed dynamic environment, and can increase the success rate of the unmanned aerial vehicle in completing the task in the high-speed moving obstacle avoidance task, so as to realize the unmanned aerial vehicle to effectively obtain state information and avoid obstacles in the high-speed dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flowchart of a reinforcement learning decoupling training method based on event features provided by an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of a virtual simulation environment provided by an embodiment of the present invention;

[0056] Figure 3 This is a flow chart of a drone test provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In order for those skilled in the art to better understand the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and the following embodiments.

[0059] like Figure 1 As shown, a reinforcement learning decoupling training method based on event features includes the following steps:

[0060] Step S1: extract offline event streams and construct training sets, where the training data includes offline event information and true value images, where the offline event information includes coordinate information, polarity, and timestamp;

[0061] Step S2: Based on the event-based variational autoencoder, the timestamps of the event sequence are normalized to build a low-dimensional feature extraction network model;

[0062] Step S3: obtaining an event frame based on the logarithmic intensity difference of image pixels at a time threshold, and constructing a high-dimensional feature extraction network model;

[0063] Step S4: Initialize a pre-trained model, i.e., a low-dimensional feature extraction network model, and pre-train the low-dimensional feature extraction network model using offline event stream information. The model consists of an event image reconstruction loss and a KL divergence loss.

[0064] Step S5: Initialize the reinforcement learning network and the high-dimensional feature extraction network model, configure training hyperparameters, and train the two network models;

[0065] Step S6: input the event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, and then perform feature splicing and fusion on the obtained low-dimensional and high-dimensional information, and use the fusion as the state information of the interaction between the reinforcement learning agent and the environment;

[0066] Step S7: UAV action space construction, specifically expressed as follows:

[0067]

[0068] θ=radians(a)

[0069] Among them, z represents the final output value after feature fusion, π is the policy network of the reinforcement learning algorithm, λ is the hyperparameter weight, the action space is first mapped to the [-1,1] interval, θ is the flight arc of the drone, which is obtained by radianizing the action a;

[0070] Step S8: construct a policy network and set a corresponding reward function. The policy network is used to obtain a motion strategy through reinforcement learning.

[0071] Step S9: Build a virtual simulation environment, construct training and test scenarios, and use the virtual simulation environment for reinforcement learning training. Configure the total number of steps for model training, learning rate, batch size, number of training rounds, and flight step length parameters. During the training process, the input event stream information will be directly input into the low-dimensional feature extraction network to obtain the potential vector Z s After aggregating the event stream information into event frames, it is input into the high-dimensional feature extraction network to obtain the latent vector Z f , Z s and Z f After splicing and fusion, it is used as the state information S. During this period, the parameters in the low-dimensional feature extraction network will not change, and the high-dimensional feature extraction network will be updated with training. Then, the state information is used to obtain the motion strategy from the action space according to the strategy network.

[0072] In step S2, using the timestamp as part of the main feature calculation will interfere with the recursive processing. How to effectively use the timestamp information is particularly critical. The present invention normalizes the timestamp by a composite rotation composite phase function. The modification of the feature by the time encoding formula is described as follows:

[0073]

[0074] Among them, z i represents the input event, Δt j，i represents the time difference between event i and event j, and τ represents the time window.

[0075] Directly using time data as input to the preprocessing neural network will interfere with feature calculation, because the global timestamp is an arbitrary value, and even each time a new event is received, the relative time difference of each event will change, requiring the feature to be recalculated. In order to better extract temporal feature information from the event stream, the time t and spatial (x, y, p) characteristics of each event e in the event stream are first separated, and then time embedding is used to normalize the timestamp to the range [0, 1], where the timestamp corresponding to the end of the window is mapped to 1. The calculation of the d-dimensional time feature of each normalized timestamp is described as follows:

[0076]

[0077] in, This formula encodes a normalized timestamp as a combination of sine / cosine values, which are then embedded into the event signature and subsequently decoded into a value indicating the recency of the event.

[0078] In step S3, the event frame is obtained based on the logarithmic intensity difference of the pixel at the time threshold, specifically:

[0079] By comparing the difference between the two RGB images, the logarithmic intensity difference of each pixel at the time threshold is calculated, and the result is compared with the threshold, and the difference in logarithmic intensity between the current frame and the previous frame is taken. The specific description is as follows:

[0080] ΔL(u, t) = log(I t )-log(I t-1 )

[0081] The polarity of each pixel is calculated based on the change threshold of the logarithmic intensity THR, as described below:

[0082]

[0083] The number of events to be triggered for each pixel is determined based on the intensity change that exceeds the threshold. Let N_max be the maximum number of events that can occur at a single pixel, as described below:

[0084]

[0085] This determines the timestamps between events, as described below:

[0086]

[0087] In step S4, a low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data. The model consists of event image reconstruction loss and KL divergence loss, which is described as follows:

[0088]

[0089] where the first term represents the reconstruction loss, P(z) is the distribution of the latent variables, and Q(z|x) is the approximate posterior of the variational autoencoder.

[0090] During training, batched data is used as input to the event-based variational autoencoder. In the specific implementation, a total of 50 batches are set, each with a batch size of 2000 events. Batches are selected by randomly sampling indices from the entire event sequence and then reading a number of subsequent events equal to the selected batch size. The Adam optimizer with a learning rate of 0.001 is used and trained for 20,000 iterations. In addition, in order to avoid the problem of vanishing KL divergence, annealing techniques are introduced. The KL loss is applied with a varying weight through the following scheduling cycle, which is also appropriately weighted according to the value of the reconstruction loss, as described specifically: Among them, β is the weight;

[0091] In step S8, the reward function of the drone is designed, including the following parts:

[0092] r t =r g，t +r c，t +r k，t +r p，t

[0093] When the drone reaches the destination, give the drone a size of r arrival The reward, r g，t is the arrival reward, expressed as follows, where p t represents the position of the drone, and g represents its destination:

[0094]

[0095] When a UAV collides, the virtual simulation environment will return the collision information collison_state according to the collision conditions. When the return state is true, the UAV will receive a collision score r. collision The punishment, r c，t Penalty for collision:

[0096]

[0097] Introducing process rewards to provide more feedback information, guide the agent to find effective behavior strategies more quickly, thereby accelerating the learning process and overcoming the difficulties caused by sparse reward problems. k，t is the process reward, expressed as follows:

[0098] r k，t =k d (||p t-1 -g||-||p t -g||)

[0099] Among them, k d Indicates the distance, p t Indicates the current position of the drone, pt-1 It indicates the position of the drone in the previous time step. When the drone is closer to the destination, it can still obtain process rewards, which encourages the drone to actively explore in the forward direction.

[0100] Adopt reward decay mechanism or reward weighting mechanism to adjust the reward weights in different states, so that the agent can pay more attention to the key points of the task or important decisions, accelerate the learning convergence and optimization process, p，t is the step penalty, which is described as follows:

[0101] r p，t = -(λ·Penalty(t))

[0102] Where Penalty(t) is a linear penalty, that is, the more steps there are in a round, the greater the penalty, and λ is a hyperparameter that controls the weight of the penalty term.

[0103] In step S8, the reinforcement learning strategy network used is the proximal policy optimization (PPO) algorithm, and the experience update method of PPO is:

[0104]

[0105] Where θ represents the policy parameters, J(θ) represents the objective function, and π is used to optimize the parameters in the objective parameters. θ represents the policy network with parameter θ, a t and t Respectively represent the action and state of the drone at time t, π θ (a t |s t ) represents the policy network π θ In s t The action taken in state a t ; R(t) represents the reward value at time t;

[0106] There are two main variants of the PPO algorithm. The first variant is based on penalties and aims to solve the overhead problem of the Trust Region Policy Optimization algorithm TRPO by incorporating constraints as penalties into the objective function. The first variant does not add separate constraints, but subtracts the KL divergence between the new and old policies from the objective function and multiplies it by a constant C. This allows the function to be optimized using simple stochastic gradient descent, as described below:

[0107]

[0108] Among them, a represents the action space of the drone, s represents the state information, and π θDenotes the policy network, A denotes the advantage function. The difficulty of the first variant is to choose a suitable value for C. The algorithm updates C according to the size of the KL divergence. If the KL divergence is too high, C increases, and if the KL divergence is too low, C decreases. The second variant is based on clipping. It does not use KL divergence in the objective function and has no constraints at all. Instead, it relies on specialized clipping of the objective function to ensure that the parameter changes are not too large. The specific description is as follows:

[0109]

[0110] Where ∈ is a hyperparameter, and clip represents the clipping function, which is used to limit the degree of update between the new and old strategies. If This means that the value of action a is higher than the average. will increase, but will not exceed 1+∈. Similarly, if will decrease, but will not exceed 1-∈.

[0111] In step S9, four obstacle avoidance scenarios are created in the AirSim virtual simulation environment, such as Figure 2 As shown in the figure, these scenes are named Test, Testlong, ChangeShape, and Train. The Train scene provides the most straightforward layout of the four scenes. Compared with the other scenes, it is narrower in width and has fewer wooden boards. The training environment is about 100 meters long, with a wall on each side, and the overall width is about 30 meters. The bottom of the axis in the environment is the center of the coordinate axis, the horizontal coordinate is set to X, the longitudinal forward coordinate is set to Y, and the height coordinate is set to Z. The initial starting point X coordinate range is randomly between [-10,10], Y is set to 0, and Z is set to 5. This scene is specially designed for training, providing more positive feedback for reinforcement learning strategies, and helping the intelligent agent to train more effectively by reducing the complexity of the scene. The Testlong scene maintains the same width as the Train scene, but significantly increases the density of wooden boards. This scene is designed to test the ability of drones to navigate in dense obstacle fields. Finally, the Changeshape scene has a similar layout to the Test scene, but replaces wooden boards with ellipsoids. This poses a huge challenge to drone models trained only in scenes featuring wooden boards, as they must adapt to the new shape and features of the ellipsoid. In each scenario, the drone takes off from a specified starting point, and the goal is to make the drone reach the destination (advance one hundred meters) while successfully avoiding dense obstacles. If the drone collides with an obstacle, the mission is considered a failure and the drone is reset to the initial starting point. In the Testlong scenario, the initial point is set at the center of the starting point, while in the other three scenarios, the starting point is randomly determined within the starting area. In addition to this, it is assumed that the drone moves at a constant predefined speed and the step size of the action is changed according to the desired frequency.

[0112] This embodiment was tested in the above three test scenarios. The test method is as follows: Figure 3 As shown, first load the trained fusion event feature network model, including the low-dimensional feature extraction network model, the high-dimensional feature extraction network model, and the reinforcement learning strategy network; then build a simulation test environment, set the drone's action space, starting point and end point, obstacle type, obstacle size, etc.; finally, perform an obstacle avoidance test on the drone, initialize the test indicators, and configure the test parameters. In each case, the drone will fly from the initial starting point and reset after successfully avoiding obstacles and reaching the destination or when colliding with an obstacle. A total of 1,000 obstacle avoidance tests were performed, and the number of successful arrivals at the destination was recorded, as shown in the following table:

[0113]

[0114] It can be seen that the solution of the present invention has good performance in extracting environmental information in a high-speed dynamic environment, and can increase the success rate of the UAV in completing the task in the high-speed moving obstacle avoidance task.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning decoupling training method based on event features, characterized in that: The method comprises: Step S1: extract offline event streams and construct training sets, where the training data includes offline event information and true value images, where the offline event information includes coordinate information, polarity, and timestamp; Step S2: Based on the event-based variational autoencoder, the timestamps of the event sequence are normalized to build a low-dimensional feature extraction network model; Step S3: obtaining an event frame based on the logarithmic intensity difference of image pixels at a time threshold, and constructing a high-dimensional feature extraction network model; Step S4: Initialize a pre-trained model, i.e., a low-dimensional feature extraction network model, and pre-train the low-dimensional feature extraction network model using offline event stream information. The model consists of an event image reconstruction loss and a KL divergence loss. Step S5: Initialize the reinforcement learning network and the high-dimensional feature extraction network model, configure training hyperparameters, and train the two network models; Step S6: input the event information into the low-dimensional feature extraction network model and the high-dimensional feature extraction network model, and then perform feature splicing and fusion on the obtained low-dimensional and high-dimensional information, and use the fusion as the state information of the interaction between the reinforcement learning agent and the environment; Step S7: UAV action space construction, specifically expressed as follows: θ=radians(a) Among them, z represents the final output value after feature fusion, π is the policy network of the reinforcement learning algorithm, λ is the hyperparameter weight, the action space is first mapped to the interval [-1, 1], θ is the flight arc of the drone, which is obtained by radianizing the action a; Step S8: construct a policy network and set a corresponding reward function. The policy network is used to obtain a motion strategy through reinforcement learning. Step S9: Build a virtual simulation environment, construct training and test scenarios, and use the virtual simulation environment for reinforcement learning training. Configure the total number of steps for model training, learning rate, batch size, number of training rounds, and flight step length parameters. During the training process, the input event stream information will be directly input into the low-dimensional feature extraction network to obtain the potential vector Z s After aggregating the event stream information into event frames, it is input into the high-dimensional feature extraction network to obtain the latent vector Z f , Z s and Z f After splicing and fusion, it is used as the state information S, and then the state information is used to obtain the motion strategy from the action space according to the strategy network.

2. The method according to claim 1, characterized in that The step S2 further comprises: The timestamps are normalized by a composite phase function with a composite rotation, and the time encoding formula modifies the features as follows: Among them, z i represents the input event, Δt j，i represents the time difference between event i and event j, and τ represents the time window; In order to better extract temporal feature information from the event stream, we first separate the temporal t and spatial (x, y, p) characteristics of each event e in the event stream, where x and y are the coordinates of the pixel and p is the polarity of the event. Then, we use time embedding to normalize the timestamp to the range [0, 1], where the timestamp corresponding to the end of the window is mapped to 1. The d-dimensional temporal feature of each normalized timestamp is calculated as follows: Where i represents the event number, t∈[0, 1], the formula encodes the normalized timestamp as a combination of sine / cosine values, which are then embedded into the event features.

3. The method according to claim 1, characterized in that: In step S3, an event frame is obtained based on the logarithmic intensity difference of pixels at a time threshold, including: By comparing the difference between the two frames of RGB images, the logarithmic intensity difference of each pixel at the time threshold is calculated, and the result is compared with the threshold, and the difference in logarithmic intensity between the current frame and the previous frame is taken, specifically: ΔL(u,t)=log(I t )-log(I t-1 ) The polarity of each pixel is calculated based on the threshold of the change in logarithmic intensity THR, specifically: According to the intensity change degree exceeding the threshold, the number of events to be excited for each pixel is determined. Let Nmax be the maximum number of events that can occur at a single pixel, specifically: This determines the timestamps between events, specifically:

4. The method according to claim 1, characterized in that: In step S4, a low-dimensional feature extraction network model is learned by maximizing the marginal log-likelihood of the training data, which consists of an event image reconstruction loss and a KL divergence loss as follows: where the first term represents the reconstruction loss, P(z) is the distribution of the latent variables, and Q(z|x) is the approximate posterior of the variational autoencoder.

5. The method according to claim 1, characterized in that In step S8, the reward function of the drone is set to: r t ＝r g,t +r c，t +r k，t +r p，t r k,t =k d (||p t-1 -g||-||p t -g||) r p,t =-(λ·Penalty(t)) Among them, r g，t To reach the reward, r c，t is the collision penalty, r k,t is the process reward, r p，t is the step penalty, p t Indicates the current position of the drone, p t-1 represents the position of the drone in the previous time step, g represents its destination, collison_state is the collision information, r collision is the collision score, k d represents the distance, Penalty(t) is a linear penalty. The more steps there are in a round, the greater the penalty. λ is a hyperparameter that controls the weight of the penalty term.

6. The method according to claim 1, characterized in that In step S8, the reinforcement learning policy network used is the proximal policy optimization PPO algorithm. There are two variants of the PPO algorithm. The first variant is based on penalty. The KL divergence deviation between the new and old strategies is subtracted from the objective function and multiplied by a constant C, allowing the function to be optimized using a simple stochastic gradient descent algorithm, as follows: Among them, a represents the action space of the drone, s represents the state information, θ represents the strategy parameter, J(θ) represents the objective function, and π θ represents the policy network, π θ (a|s) represents the policy network π θ The action taken in state s is a, and A represents the advantage function; The second variant is based on clipping, which relies on clipping the objective function to ensure that the changes in parameters are not too large, as follows: Where ∈ is a hyperparameter, and clip represents the clipping function, which is used to limit the degree of update between the new and old strategies. If This means that the value of action a is higher than the average. will increase, but will not exceed 1+∈; if will decrease, but will not exceed 1-∈.

Images