A local anti-pursuit method for surface unmanned vehicles based on aggregation of spatiotemporal features

Abstract

The present invention discloses a local anti-pursuit method for surface unmanned boats that aggregates spatiotemporal features. The current local navigation algorithm for surface unmanned boats mainly carries out local collision avoidance path planning for static and uniform linear motion dynamic obstacles. Due to loss of control or malicious blocking and other behaviors, the unmanned boat may encounter pursuit, which poses a great threat to its navigation safety. The present invention proposes an anti-pursuit path planning method suitable for complex water surface environments, including: simulating multi-source shipborne sensors to generate simulation signals to provide environmental perception data, constructing a coding structure that aggregates spatiotemporal features to distinguish between dynamic and static obstacles in the scene, and using a value-distributed reinforcement learning algorithm to build an end-to-end navigation path planning model. The present invention can identify the motion characteristics of obstacles in the environment, thereby enhancing the decision-making ability and navigation robustness of the unmanned boat in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of intelligent unmanned ships, and in particular to an end-to-end unmanned boat local navigation planning method that aggregates spatiotemporal features. Specifically, the present invention relates to a method that extracts environmental observation information using a spatiotemporal feature coding structure as input to a deep reinforcement learning algorithm, thereby guiding the unmanned boat to autonomously allocate weight information to achieve local obstacle avoidance. Background Art

[0002] Unmanned surface boats have been the research focus in the field of smart ships and intelligent shipping in recent years. Improving the autonomous navigation capabilities of ships is inseparable from their own real-time path planning capabilities and highly reliable obstacle avoidance capabilities. In the past five years, a large number of emerging research has emerged in the field of ship path planning. These works focus on designing different navigation scenarios to verify the path planning capabilities of decision-making algorithms. Global path planning, researchers obtain static environmental features through information such as electronic nautical charts, and plan a route from the starting point to the end point. This type of algorithm only needs to adjust the trajectory when the ship deviates significantly from the preset path, and there is no need to update the planned path in real time. Local path planning, using real-time information about surrounding obstacles to guide the changes in the speed and heading angle of the ship at the next moment, this algorithm needs to perceive environmental information in real time and make reasonable motion decisions.

[0003] This paper focuses on local path planning, exploring local anti-pursuit methods for unmanned vehicles in complex eddy current environments, such as those with highly dynamic pursuers and high-density obstacles. It also investigates end-to-end robust ship motion decision-making techniques to achieve real-time, highly reliable autonomous collision avoidance and navigation planning for unmanned vehicles in scenarios without prior information. Summary of the Invention

[0004] Technical problem: The technical problem to be solved by the present invention is to provide a robust navigation planning method for surface unmanned boats that aggregates spatiotemporal characteristics, in response to the local path planning direction involved in the background technology, and to achieve real-time collision avoidance of ships with obstacles in various motion modes, targeting high-dynamic pursuers, high-density obstacles and eddy-interfered water environments.

[0005] Technical solution: To solve the above technical problems, the present invention adopts a local anti-pursuit method for surface unmanned boats that aggregates spatiotemporal features. The method includes:

[0006] Step 1: Define the local navigation area, set the starting and ending coordinates of the surface unmanned vehicle in the area, and randomly generate eddy current signals in the area;

[0007] Step 2: Construct a navigation environment and randomly generate static obstacles of varying sizes and dynamic obstacles of random positions in the blank area.

[0008] Step 3: Based on the sensor types carried by the real ship, a simulated signal is generated that provides information on speed, ship position, and relative distance to obstacles, and the observation signal is constructed from this information.

[0009] Step 4: The observation signal is fed into the spatiotemporal feature aggregation module to aggregate spatial features along the time dimension in a frame-by-frame stacking manner; and the observation signal is fed into the continuous-time neural network frame by frame to aggregate temporal features;

[0010] Step 5: Determine the distributed reinforcement learning algorithm as the backbone framework of the navigation planning network and define the action space and reward function;

[0011] Step 6: Use the experience pool to collect experience data during training, optimize network weight parameters with small batches of experience data, and train the deep reinforcement learning model with the experience replay strategy until convergence;

[0012] Step 7: Build an experimental verification scenario, load the converged model parameters, and verify the ship's anti-pursuit and collision avoidance capabilities in the local environment and its generalization capabilities to cope with new scenarios.

[0013] The local navigation area is a simulation environment built to verify the effectiveness of the algorithm.

[0014] The spatiotemporal feature aggregation module is composed of an independent spatial feature aggregation module and a temporal feature extraction module connected in series.

[0015] The step 1 specifically includes the following steps:

[0016] 1-1: Define the local navigation area of ​​the surface unmanned vehicle, set the length to H and the width to W;

[0017] 1-2: Initialize the starting coordinates P of the surface unmanned vehicle STA (x,y), end point coordinates P END (x, y), within the current local navigation area, the coordinates of the unmanned boat at each moment are P(x, y).

[0018] The step 2 specifically includes the following steps:

[0019] 2-1: Determine the random operator and randomly initialize the position coordinates of static obstacles in the blank area Radius size Randomly selected according to the mean distribution; s i represents the i-th static obstacle;

[0020] 2-2: For dynamic obstacles, randomly generate their initial position coordinates in the remaining blank area The end point is set as the position coordinate P(x,y) of the surface unmanned boat at the current moment, that is, the end point changes with the movement of the unmanned boat. jrepresents the jth dynamic obstacle.

[0021] The step 3 comprises the following steps:

[0022] 3-1: Based on the types of ship-borne sensors, simulate the working principles and output characteristics of each sensor to generate the current observation signal;

[0023] 3-2: Generate satellite and inertial device signals to provide the ship's own position P(x,y);

[0024] 3-3: Simulate and generate Doppler speed meter signal to provide the ship's current ground speed V(v x ,v y ).

[0025] 3-4: Simulate and generate laser radar signals to provide the relative distance information D (d1, d2, ..., d N ), where N is the number of laser beams generated when simulating the laser radar signal;

[0026] 3-5: Combine the ship's own position information, speed information and obstacle distance information to use as the observation signal s in the current environment t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N )}.

[0027] The step 4 comprises the following steps:

[0028] 4-1: Observation signal s spliced ​​together in step 3 t , design a stack structure, stack the signals along the time dimension, and construct spatial feature information S rich in the spatial position changes of obstacles t ={s t-k ,…,s t-1 ,s t}, where s t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N )}; A rectangular convolutional neural network is used to extract motion features from the spatial feature information.

[0029] 4-2: Based on the feature aggregation of spatial feature information using neural networks, a continuous-time neural network is used to further extract its internal motion-dependent characteristics and obtain the identification characteristics of environmental observation signals in the time dimension; specifically, the continuous-time neural network is composed of sub-networks {f, g, h}, and the three sub-networks share the basic skeleton network structure.

[0030] The step 5 comprises the following steps:

[0031] 5-1: The Deterministic Value Distributed Reinforcement Learning (IQN) algorithm is a network framework that models the distribution of cumulative returns and uses this information to guide decision-making, avoiding the use of mean return information in all states while ignoring changes in risk in the scenario.

[0032] 5-2: Define the action space as the change of acceleration and angular velocity in discrete space α is the acceleration, is the angular velocity;

[0033] 5-3: Determine the reward function r t , define the reward factor for reaching the end point and the penalty factor for colliding with obstacles, define the time penalty factor, and define the end point g guidance reward function at the current time t

[0034] The step 6 comprises the following steps:

[0035] 6-1: Initialize the deep reinforcement learning network and initialize the input state s t , select action a in the current input state t , execute this action to get the current moment reward value r t And the next moment state s t+1 , the experience bar (s t , a t ,r t ,s t+1 ) is stored in the experience replay pool.

[0036] 6-2: Determine the total number of experience replay pools. When the cumulative number of training experience bars exceeds this total number, extract experience bar data in small batches. Use the gradient descent algorithm to calculate the gradient during the backpropagation process to complete the network parameter update until the model converges.

[0037] The step 7 comprises the following steps:

[0038] 7-1: The properties of vortices, static obstacles, and dynamic obstacles in the experimental scene remain consistent with those in the training scene, but their positions and numbers are randomly initialized according to the experimental requirements.

[0039] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution, which has the following advantages:

[0040] This paper proposes a local anti-pursuit method for surface unmanned vehicles (UAVs) that aggregates spatiotemporal features. This method constructs an end-to-end local path planning network for the UAV, from sensor-generated environmental data to the UAV's navigation planning and decision signals. This network fully extracts the spatial position changes of continuously observed signals and, through a continuous-time network, further extracts the long-term motion dependencies of obstacles contained in these features. This method enables local collision avoidance and anti-pursuit navigation route planning in high-risk, obstacle-dense scenarios.

[0041] 2. This invention fully considers the water surface environment and the characteristics of shipboard sensors, adding eddy current interference to simulate the impact of the dynamic water surface environment on the ship's speed. The output characteristics of shipboard sensors are simulated to generate environmental observation data, ensuring that the simulation results are consistent with the ship's front-end sensor input in the real environment. Compared to current algorithms that rely heavily on obstacle location and body length information to infer the location, the lidar and other sensor data sources used in this invention have a higher update frequency, and the errors transmitted during the obstacle position solution are smaller, which helps to generate more robust and real-time decision signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a local anti-pursuit method for an unmanned surface boat that aggregates spatiotemporal characteristics according to the present invention.

[0043] Figure 2 It is a schematic diagram of the environment observation state vector constructed by the simulated shipborne sensor of the present invention.

[0044] Figure 3 This is a diagram of the value-distributed reinforcement learning network architecture for aggregating spatiotemporal features of the present invention.

[0045] Figure 4 This is a rendering of the local anti-pursuit and navigation path planning of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0047] like Figure 1 As shown in FIG, the present invention discloses a local anti-pursuit method for a surface unmanned boat that aggregates spatiotemporal features. In the local path planning process, the working principle and output state of the simulated sensor are used to generate environmental observation signals under multiple information sources, and a stack structure is used to stack environmental observations along the time dimension to obtain state information. Figure 3The network architecture shown in the figure includes spatial and temporal feature extraction modules, and a fusion distributed reinforcement learning algorithm to achieve anti-pursuit guidance and navigation path planning for unmanned surface vessels in high-risk environments. The proposed method mainly serves the navigation path planning module of unmanned surface vessels, providing a real-time and highly secure path planning algorithm for unmanned surface vessels in highly dynamic water surface and strong interference environments. Specifically, it includes the following steps:

[0048] Step 1: Define the scope of the local navigation area, set the starting and ending coordinates of the surface unmanned vehicle in the area, and randomly generate eddy current signals in the area.

[0049] Step 2: Construct a navigation environment and randomly generate static obstacles of varying sizes and dynamic obstacles of random positions in the blank area.

[0050] Step 3: Based on the type of sensors carried by the real ship, generate a simulated signal that provides information on speed, the ship's own position, and the relative distance to obstacles, and use this to construct the observation signal.

[0051] Step 4: For the observation signal, aggregate spatial features along the time dimension in an inter-frame stacking manner; and input the observation signal frame by frame into the continuous-time neural network to aggregate temporal features.

[0052] Step 5: Determine the distributed reinforcement learning IQN algorithm as the backbone framework of the navigation planning network and define the action space and reward function.

[0053] Step 6: Use the experience pool to collect experience data during the training process, optimize the network weight parameters with small batches of experience data, and train the deep reinforcement learning model with the experience replay strategy until convergence.

[0054] Step 7: Build an experimental verification scenario, load the converged model parameters, and verify the ship's anti-pursuit and collision avoidance capabilities in the local environment and its generalization capabilities to cope with new scenarios.

[0055] Furthermore, step 1 specifically includes the following steps:

[0056] 1-1: Define the local navigation area of ​​the surface unmanned vehicle, and set the length value to H and the width value to W.

[0057] 1-2: Initialize the starting coordinates P of the surface unmanned vehicle STA (x,y), end point coordinates P END (x, y). Within the current local navigation area, the coordinates of the unmanned boat at each moment are P(x, y).

[0058] Furthermore, step 2 specifically includes the following steps:

[0059] 2-1: Determine the random operator and randomly initialize the position coordinates of static obstacles in the blank area Radius size Randomly selected from a mean distribution.

[0060] 2-2: For dynamic obstacles, randomly generate their initial position coordinates in the remaining blank area The end point is set as the position coordinate P(x, y) of the surface unmanned boat at the current moment, that is, the end point changes continuously with the movement of the unmanned boat.

[0061] Furthermore, step 3 includes the following steps:

[0062] 3-1: Based on the type of ship-borne sensors, simulate the working principle and output characteristics of each sensor to generate the current observation signal.

[0063] 3-2: Generate satellite and inertial device signals to provide the ship's own position P(x,y).

[0064] 3-3: Simulate and generate Doppler speed meter signal to provide the ship's current ground speed V(v x ,v y ).

[0065] 3-4: Simulate and generate laser radar signals to provide the relative distance information D (d1, d2, ..., d N ), where N is the number of laser beams generated when simulating the lidar signal.

[0066] 3-5: Combine the ship's own position information, speed information and obstacle distance information to use as the observation signal s in the current environment t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N )}.

[0067] Furthermore, step 4 includes the following steps:

[0068] 4-1: Observation signal s spliced ​​together in step 3 t , design a stack structure, stack the signals along the time dimension, and construct spatial feature information S rich in the spatial position changes of obstacles t ={s t-k ,…,s t-1 ,s t}, where s t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N)}. A rectangular convolutional neural network is used to extract motion features from the spatial feature information.

[0069] 4-2: Based on the feature aggregation of spatial feature information using neural networks, a continuous-time neural network is used to further extract its internal motion-dependent characteristics and obtain the identification characteristics of environmental observation signals in the time dimension. Specifically, the continuous-time neural network is composed of sub-networks {f, g, h}, and the three sub-networks share the basic skeleton network structure.

[0070] Furthermore, step 5 includes the following steps:

[0071] 5-1: Deterministic value distributed reinforcement learning IQN algorithm is a network framework that models the distribution information of cumulative returns and uses the return distribution information to guide decision-making, avoiding using the mean return information in all states and ignoring the risk changes in the scenario.

[0072] 5-2: Define the action space as the change of acceleration and angular velocity in discrete space

[0073] 5-3: Determine the reward function r t , define the reward factor for reaching the end point and the penalty factor for colliding with obstacles, define the time penalty factor, and define the end point guidance reward function at the current moment

[0074] Furthermore, step 6 includes the following steps:

[0075] 6-1: Initialize the deep reinforcement learning network and initialize the input state s t , select action a in the current input state t , execute this action to get the current moment reward value r t And the next moment state s t+1 , the experience bar (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool.

[0076] 6-2: Determine the total number of experience replay pools. When the cumulative number of training experience bars exceeds this total number, extract experience bar data in small batches. Use the gradient descent algorithm to calculate the gradient during the backpropagation process to complete the network parameter update until the model converges.

[0077] Furthermore, step 7 includes the following steps:

[0078] 7-1: The properties of vortices, static obstacles, and dynamic obstacles in the experimental scene remain consistent with those in the training scene, but their positions and numbers are randomly initialized according to the experimental requirements.

[0079] Example 1

[0080] This invention relates to local obstacle avoidance technology for unmanned surface vessels, aiming to guide them in autonomously planning navigation paths in dense, dynamic, and high-risk water environments. In this embodiment, a deep reinforcement learning network model is trained in a simulated environment. The algorithm's effectiveness is verified in a single scenario, demonstrating the specific steps involved in local anti-pursuit and path planning for the unmanned vessel. The relevant parameter configuration and operational procedures are as follows:

[0081] Step 1: Define a local obstacle avoidance area of ​​50 x 50 m. Set the starting point of the UAV to (5, 5) and the ending point to (45, 45). The UAV is 1.255 m long and 0.29 m wide. Randomly initialize the vortex position.

[0082] Step 2: Randomly initialize static obstacles in the blank area, with the obstacle radius uniformly distributed between [1,3] m. Randomly initialize the starting coordinates of the dynamic pursuer obstacle, with the pursuer ship having a length of 1 m and a width of 0.5 m.

[0083] Step 3: Build Figure 2 The composite observation vector shown contains the ship's position provided by satellite and inertial device signals, velocity information obtained from the Doppler velocimeter, and relative distance information obtained from the lidar. The stack length is set to 4, and four consecutive frames of observation vectors are stacked as input to the deep reinforcement learning network.

[0084] Step 4: Send the composite observation signal to Figure 2 In the constructed spatial feature extraction and temporal feature extraction modules, the data dimension is reduced while the spatiotemporal features of the environmental observation signals are aggregated, and the action signals are output through the value distribution reinforcement learning network decision.

[0085] Step 5: The action space contains the acceleration change in discrete space [-0.2, 0, 0.2] m / s 2 , angular velocity changes to [-5, 0, 5]° / s. In the reward function, the reward value for reaching the target is set to 50, the penalty value for collision is set to -100, the time penalty is set to -1, and the difference between the distance to the end point at the previous moment and the distance to the end point at the current moment is used as the sub-reward factor.

[0086] Step 6: Use the experience revisit strategy to train the neural network until convergence.

[0087] Step 7: Load the network parameters trained to convergence in step 6 into the model, randomly generate a local navigation environment, and verify the anti-pursuit ability and navigation path planning ability of the ship in the local environment. Figure 4The experimental results are shown in Figure 2. The orange rectangle represents the dynamic pursuer, the green rectangle represents the surface unmanned vehicle, and the gray circle represents the static obstacle.

[0088] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such, will not be interpreted in an idealized or overly formal sense.

[0089] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A local anti-pursuit method for surface unmanned vehicles that aggregates spatiotemporal features, characterized in that: The method includes: Step 1: Define the local navigation area, set the starting and ending coordinates of the surface unmanned vehicle in the area, and randomly generate eddy current signals in the area; Step 2: Construct a navigation environment and randomly generate static obstacles of varying sizes and dynamic obstacles of random positions in the blank area. Step 3: Based on the sensor types carried by the real ship, a simulated signal is generated that provides information on speed, ship position, and relative distance to obstacles, and the observation signal is constructed from this information. Step 4: The observation signal is fed into the spatiotemporal feature aggregation module to aggregate spatial features along the time dimension in a frame-by-frame stacking manner; and the observation signal is fed into the continuous-time neural network frame by frame to aggregate temporal features; Step 5: Determine the distributed reinforcement learning algorithm as the backbone framework of the navigation planning network and define the action space and reward function; Step 6: Use the experience pool to collect experience data during training, optimize network weight parameters with small batches of experience data, and train the deep reinforcement learning model with the experience replay strategy until convergence; Step 7: Build an experimental verification scenario, load the converged model parameters, and verify the ship's anti-pursuit and collision avoidance capabilities in the local environment and its generalization capabilities to cope with new scenarios.

2. The local anti-pursuit method for unmanned surface vehicles based on the aggregation of spatiotemporal features according to claim 1 is characterized in that: The local navigation area is a simulation environment built to verify the effectiveness of the algorithm.

3. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The spatiotemporal feature aggregation module is composed of an independent spatial feature aggregation module and a temporal feature extraction module connected in series.

4. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 1 specifically includes the following steps: 1-1: Define the local navigation area of ​​the surface unmanned vehicle, set the length to H and the width to W; 1-2: Initialize the starting coordinates P of the surface unmanned vehicle STA (x,y), end point coordinates P END (x, y), within the current local navigation area, the coordinates of the unmanned boat at each moment are P(x, y).

5. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 2 specifically includes the following steps: 2-1: Determine the random operator and randomly initialize the position coordinates of static obstacles in the blank area Radius size Randomly selected according to the mean distribution; s i represents the i-th static obstacle; 2-2: For dynamic obstacles, randomly generate their initial position coordinates in the remaining blank area The end point is set as the position coordinate P(x,y) of the surface unmanned boat at the current moment, that is, the end point changes with the movement of the unmanned boat. j represents the jth dynamic obstacle.

6. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 3 comprises the following steps: 3-1: Based on the types of ship-borne sensors, simulate the working principles and output characteristics of each sensor to generate the current observation signal; 3-2: Generate satellite and inertial device signals to provide the ship's own position P(x,y); 3-3: Simulate and generate Doppler speed meter signal to provide the ship's current ground speed V(v x ,v y ); 3-4: Simulate and generate laser radar signals to provide the relative distance information D (d1, d2, ..., d N ), where N is the number of laser beams generated when simulating the laser radar signal; 3-5: Combine the ship's own position information, speed information and obstacle distance information to use as the observation signal s in the current environment t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N )}.

7. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 4 comprises the following steps: 4-1: Observation signal s spliced ​​together in step 3 t , design a stack structure, stack the signals along the time dimension, and construct spatial feature information S rich in the spatial position changes of obstacles t ={s t-k ,…,s t-1 ,s t }, where s t = {P(x,y),V(v x ,v y ),D(d1,d2,…,d N )}; A rectangular convolutional neural network is used to extract motion features from the spatial feature information; 4-2: Based on the feature aggregation of spatial feature information using neural networks, a continuous-time neural network is used to further extract its internal motion-dependent characteristics and obtain the identification characteristics of environmental observation signals in the time dimension; specifically, the continuous-time neural network is composed of sub-networks {f, g, h}, and the three sub-networks share the basic skeleton network structure.

8. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 5 comprises the following steps: 5-1: The Deterministic Value Distributed Reinforcement Learning (IQN) algorithm is a network framework that models the distribution of cumulative returns and uses this information to guide decision-making, avoiding the use of mean return information in all states while ignoring changes in risk in the scenario. 5-2: Define the action space as the acceleration and angular velocity changes a in discrete space t ={α,ω}; α is the acceleration, is the angular velocity; 5-3: Determine the reward function r t , define the reward factor for reaching the end point and the penalty factor for colliding with obstacles, define the time penalty factor, and define the end point g guidance reward function at the current time t 9. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1 is characterized in that: The step 6 comprises the following steps: 6-1: Initialize the deep reinforcement learning network and initialize the input state s t , select action a in the current input state t , execute this action to get the current moment reward value r t and the next moment state s t+1 , the experience bar (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool; 6-2: Determine the total number of experience replay pools. When the cumulative number of training experience bars exceeds this total number, extract experience bar data in small batches. Use the gradient descent algorithm to calculate the gradient during the backpropagation process to complete the network parameter update until the model converges.

10. The local anti-pursuit method for unmanned surface vehicles by aggregating spatiotemporal features according to claim 1, characterized in that: Step 7 The following steps are involved: 7-1: The properties of vortices, static obstacles, and dynamic obstacles in the experimental scene remain consistent with those in the training scene, but their positions and numbers are randomly initialized according to the experimental requirements.

Images